nn

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 10, 2021 License: BSD-2-Clause Imports: 14 Imported by: 0

README

Neural Network

The inclusion of neural models in the nn sub-package is mostly arbitrary.

Not all neural models are useful. For instance, I wanted to implement many recurrent networks for the sake of curiosity, but in the end, the LSTM and GRU almost always gave us the best performance in natural language processing tasks.

We might decide - based on your suggestions - to delete some of them to lighten the core package.

Documentation

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Affine 

func Affine(g *ag.Graph, xs ...ag.Node) ag.Node

Affine performs an affine transformation over an arbitrary (odd) number of nodes held in the input. The first node is the “bias”, which is added to the output as-is. The remaining nodes of the form "Wx" are multiplied together in pairs, then added. The pairs except the first whose "x" is nil are not considered. y = b + W1x1 + W2x2 + ... + WnXn

func BiAffine 

func BiAffine(g *ag.Graph, w, u, v, b, x1, x2 ag.Node) ag.Node

BiAffine performs a biaffine transformation.

func BiLinear 

func BiLinear(g *ag.Graph, w, x1, x2 ag.Node) ag.Node

BiLinear performs a bilinear transformation of the type (x_1 W x_2)

func ClearSupport 

func ClearSupport(m Model)

ClearSupport clears the support structure of all model's parameters (including sub-params).

func Conv2D 

func Conv2D(g *ag.Graph, w, x ag.Node, xStride, yStride int) ag.Node

Conv2D performs a 2D convolution.

func DumpParamsVector 

func DumpParamsVector(model Model) *mat.Dense

DumpParamsVector dumps all params of a Model into a single Dense vector.

func ForEachParam 

func ForEachParam(m Model, callback func(param Param))

ForEachParam iterate all the parameters of a model also exploring the sub-parameters recursively.

func ForEachParamStrict 

func ForEachParamStrict(m Model, callback func(param Param))

ForEachParamStrict iterate all the parameters of a model without exploring the sub-models.

func LoadParamsVector 

func LoadParamsVector(model Model, vector *mat.Dense)

LoadParamsVector sets all params of a Model from a previously dumped Dense vector.

func PayloadMarshalBinaryTo 

func PayloadMarshalBinaryTo(supp *Payload, w io.Writer) (int, error)

PayloadMarshalBinaryTo returns the number of bytes written into w and an error, if any.

func Separate 

func Separate(g *ag.Graph, x ag.Node) [][]ag.Node

Separate returns a matrix of Node(s) represented as a slice of slice containing the elements extracted from the input. The dimensions of the resulting matrix are the same of the input.

func SeparateVec 

func SeparateVec(g *ag.Graph, x ag.Node) []ag.Node

SeparateVec returns a slice of Node(s) containing the elements extracted from the input. The size of the vector equals the number of input elements. You can think of this method as the inverse of the ag.Concat operator.

func SplitVec 

func SplitVec(g *ag.Graph, x ag.Node, chunks int) []ag.Node

SplitVec splits the x Node into multiple chunks. It panics if the x Node is not a vector. TODO: optimize, this is extremely inefficient!

func ToNode added in v0.2.0 

func ToNode(i interface{}) ag.Node

ToNode coerces the given value to an ag.Node.

If the underlying value is already of type ag.Node, it is simply returned unmodified. If the value is a slice of nodes []ag.Node which contains exactly one item, the method returns just that node. If the value is a slice of zero, two or more nodes, the method panics. In case of any other type, the method panics as well.

Example 
package main

import (
	"fmt"
	"github.com/nlpodyssey/spago/pkg/ml/ag"
	"github.com/nlpodyssey/spago/pkg/ml/nn"
)

func main() {
	g := ag.NewGraph()

	a := g.NewScalar(1)
	b := []ag.Node{g.NewScalar(2)}

	fmt.Println(nn.ToNode(a).Value())
	fmt.Println(nn.ToNode(b).Value())

}

Output:

[1]
[2]

func ToNodes added in v0.2.0 

func ToNodes(i interface{}) []ag.Node

ToNodes coerces the given value to a slice of nodes []ag.Node.

If the value is already of type []ag.Node, it is simply returned unmodified. If the value is a single ag.Node, the method returns a new slice of nodes, containing just that node. In case of any other type, the method panics.

Example 
package main

import (
	"fmt"
	"github.com/nlpodyssey/spago/pkg/ml/ag"
	"github.com/nlpodyssey/spago/pkg/ml/nn"
)

func main() {
	g := ag.NewGraph()

	a := g.NewScalar(1)
	b := []ag.Node{g.NewScalar(2), g.NewScalar(3)}

	fmt.Printf("len: %d, first value: %v\n", len(nn.ToNodes(a)), nn.ToNodes(a)[0].Value())
	fmt.Printf("len: %d, first value: %v\n", len(nn.ToNodes(b)), nn.ToNodes(b)[0].Value())

}

Output:

len: 1, first value: [1]
len: 2, first value: [2]

func ZeroGrad 

func ZeroGrad(m Model)

ZeroGrad set the gradients of all model's parameters (including sub-params) to zeros.

Types

type BaseModel added in v0.2.0 

type BaseModel struct {
	// Context
	Ctx Context
}

BaseModel satisfies some methods of the Model interface. Don't use it directly; it is meant to be embedded in other processors to reduce the amount of boilerplate code.

func (*BaseModel) Graph added in v0.2.0 

func (m *BaseModel) Graph() *ag.Graph

Graph returns the computational graph on which the (reified) model operates. It panics if the Graph is nil.

func (*BaseModel) InitProcessor added in v0.2.0 

func (m *BaseModel) InitProcessor()

InitProcessor is used to initialize structures and data useful for the Forward(). nn.Reify() automatically invokes InitProcessor() for any sub-models.

func (*BaseModel) IsProcessor added in v0.2.0 

func (m *BaseModel) IsProcessor() bool

IsProcessor returns whether the model has been reified (i.e., contextualized to operate on a graph) and can perform the Forward().

func (*BaseModel) Mode added in v0.2.0 

func (m *BaseModel) Mode() ProcessingMode

Mode returns whether the (reified) model is being used for training or inference.

type Context 

type Context struct {
	// Graph is the computational graph on which the processor(s) operate.
	Graph *ag.Graph
	// Mode regulates the different usage of some operations whether you're doing training or inference.
	Mode ProcessingMode
}

Context is used to reify a Model (inc. sub-models) to operate on a graph, according to the desired ProcessingMode.

func (*Context) MarshalBinary added in v0.3.0 

func (c *Context) MarshalBinary() ([]byte, error)

MarshalBinary satisfies package pkg/encoding/gob custom marshaling interface We never want to marshal Context, hence this implementation returns an empty value

func (*Context) UnmarshalBinary added in v0.3.0 

func (c *Context) UnmarshalBinary(data []byte) error

UnmarshalBinary satisfies pkg/encoding/gob custom marshaling interface

type DefaultParamsIterator 

type DefaultParamsIterator struct {
	// contains filtered or unexported fields
}

DefaultParamsIterator is spaGO default implementation of a ParamsGetter.

func NewDefaultParamsIterator 

func NewDefaultParamsIterator(models ...Model) *DefaultParamsIterator

NewDefaultParamsIterator returns a new DefaultParamsIterator.

func (*DefaultParamsIterator) Params added in v0.2.0 

func (i *DefaultParamsIterator) Params() []Param

Params returns a slice with all Param elements from all models held by the DefaultParamsIterator.

type Model 

type Model interface {
	// Graph returns the computational graph on which the model operates (can be nil).
	Graph() *ag.Graph
	// Mode returns whether the model is being used for training or inference.
	Mode() ProcessingMode
	// IsProcessor returns whether the model has been reified (i.e., contextualized to operate
	// on a graph) and can perform the Forward().
	IsProcessor() bool
	// InitProcessor is used to initialize structures and data useful for the Forward().
	// nn.Reify() automatically invokes InitProcessor() for any sub-models.
	InitProcessor()
}

Model is implemented by all neural network architectures. You can assign parameters (i.e. nn.Param) as regular attributes (if any). A Model can also contain other Model(s), allowing to nest them in a tree structure. Through "reification" (i.e. nn.Reify()), a Model operates as a "processor" using the computational graph. The Forward() operation can only be performed on a reified model (a.k.a. processor).

func MakeNewModels 

func MakeNewModels(n int, callback func(i int) Model) []Model

MakeNewModels return n new models. The callback is delegated to return a new model for each i-item.

func Reify added in v0.2.0 

func Reify(ctx Context, m Model) Model

Reify returns a new "reified" model (a.k.a. processor) to execute the forward step.

type Param 

type Param interface {
	ag.Node // it implies fn.Operand and ag.GradValue too

	// Name returns the params name (can be empty string).
	Name() string
	// SetName set the params name (can be empty string).
	SetName(name string)
	// Type returns the params type (weights, biases, undefined).
	Type() ParamsType
	// SetType set the params type (weights, biases, undefined).
	SetType(pType ParamsType)
	// SetRequiresGrad set whether the param requires gradient, or not.
	SetRequiresGrad(value bool)
	// ReplaceValue replaces the value of the parameter and clears the support structure.
	ReplaceValue(value mat.Matrix)
	// ApplyDelta updates the value of the underlying storage applying the delta.
	ApplyDelta(delta mat.Matrix)
	// Payload returns the optimizer support structure (can be nil).
	Payload() *Payload
	// SetPayload is a thread safe operation to set the given Payload on the
	// receiver Param.
	SetPayload(payload *Payload)
	// ClearPayload clears the support structure.
	ClearPayload()
}

Param is the interface for a Model parameter.

func NewParam 

func NewParam(value mat.Matrix, opts ...ParamOption) Param

NewParam returns a new param.

type ParamOption 

type ParamOption func(*param)

ParamOption allows to configure a new Param with your specific needs.

func RequiresGrad 

func RequiresGrad(value bool) ParamOption

RequiresGrad is an option to specify whether a Param should be trained or not.

func SetStorage 

func SetStorage(storage kvdb.KeyValueDB) ParamOption

SetStorage is an option to specify a kvdb.KeyValueDB storage. This is useful, for example, for a memory-efficient embeddings Param implementation.

type ParamSerializer 

type ParamSerializer struct {
	// contains filtered or unexported fields
}

ParamSerializer allows serialization and deserialization of a single Param.

func NewParamSerializer added in v0.2.0 

func NewParamSerializer(p Param) (*ParamSerializer, error)

NewParamSerializer returns a new ParamSerializer. It returns an error if the Param doesn't support serialization.

func (ParamSerializer) ApplyDelta added in v0.2.0 

func (r ParamSerializer) ApplyDelta(delta mat.Matrix)

ApplyDelta updates the value of the underlying storage applying the delta.

func (ParamSerializer) ClearPayload added in v0.2.0 

func (r ParamSerializer) ClearPayload()

ClearPayload clears the support structure.

func (*ParamSerializer) Deserialize 

func (s *ParamSerializer) Deserialize(r io.Reader) (n int, err error)

Deserialize assigns reads a Param the reader.

func (ParamSerializer) Grad added in v0.2.0 

func (r ParamSerializer) Grad() mat.Matrix

Grad returns the gradients accumulated during the backward pass.

func (ParamSerializer) Graph added in v0.2.0 

func (r ParamSerializer) Graph() *ag.Graph

Graph returns always nil since the "pure" parameter is not associated with any graph.

func (ParamSerializer) HasGrad added in v0.2.0 

func (r ParamSerializer) HasGrad() bool

HasGrad returns true if there are accumulated gradients.

func (ParamSerializer) ID added in v0.2.0 

func (r ParamSerializer) ID() int

ID returns always -1 since the "pure" parameter is not associated with any graph.

func (ParamSerializer) MarshalBinary added in v0.2.0 

func (r ParamSerializer) MarshalBinary() ([]byte, error)

MarshalBinary satisfies package pkg/encoding/gob custom marshaling interface

func (ParamSerializer) Name added in v0.2.0 

func (r ParamSerializer) Name() string

Name returns the params name (can be empty string).

func (ParamSerializer) Payload added in v0.2.0 

func (r ParamSerializer) Payload() *Payload

Payload returns the optimizer support structure (can be nil).

func (ParamSerializer) PropagateGrad added in v0.2.0 

func (r ParamSerializer) PropagateGrad(grad mat.Matrix)

PropagateGrad accumulate the gradients

func (ParamSerializer) ReplaceValue added in v0.2.0 

func (r ParamSerializer) ReplaceValue(value mat.Matrix)

ReplaceValue replaces the value of the parameter and clears the support structure.

func (ParamSerializer) RequiresGrad added in v0.2.0 

func (r ParamSerializer) RequiresGrad() bool

RequiresGrad returns true if the param requires gradients.

func (ParamSerializer) ScalarValue added in v0.2.0 

func (r ParamSerializer) ScalarValue() mat.Float

ScalarValue returns the the scalar value of the node. It panics if the value is not a scalar. Note that it is not possible to start the backward step from a scalar value.

func (*ParamSerializer) Serialize 

func (s *ParamSerializer) Serialize(w io.Writer) (int, error)

Serialize dumps the Param to the writer.

func (ParamSerializer) SetName added in v0.2.0 

func (r ParamSerializer) SetName(name string)

SetName set the params name (can be empty string).

func (ParamSerializer) SetPayload added in v0.2.0 

func (r ParamSerializer) SetPayload(payload *Payload)

SetPayload is a thread safe operation to set the given Payload on the receiver Param.

func (ParamSerializer) SetRequiresGrad added in v0.2.0 

func (r ParamSerializer) SetRequiresGrad(value bool)

RequiresGrad is an option to specify whether a Param should be trained or not.

func (ParamSerializer) SetType added in v0.2.0 

func (r ParamSerializer) SetType(pType ParamsType)

SetType set the params type (weights, biases, undefined).

func (ParamSerializer) TimeStep added in v0.2.0 

func (r ParamSerializer) TimeStep() int

TimeStep returns always 0 since the "pure" parameter is not associated with any graph.

func (ParamSerializer) Type added in v0.2.0 

func (r ParamSerializer) Type() ParamsType

Type returns the params type (weights, biases, undefined).

func (ParamSerializer) UnmarshalBinary added in v0.2.0 

func (r ParamSerializer) UnmarshalBinary(data []byte) error

UnmarshalBinary satisfies pkg/encoding/gob custom marshaling interface

func (ParamSerializer) Value added in v0.2.0 

func (r ParamSerializer) Value() mat.Matrix

Value returns the value of the delegate itself.

func (ParamSerializer) ZeroGrad added in v0.2.0 

func (r ParamSerializer) ZeroGrad()

ZeroGrad clears the gradients.

type Params added in v0.2.0 

type Params []Param

Params extends a slice of Param with Nodes() method.

func (Params) Nodes added in v0.2.0 

func (ps Params) Nodes() []ag.Node

Nodes converts the slice of Param into a slice of ag.Node.

type ParamsGetter added in v0.2.0 

type ParamsGetter interface {
	Params() []Param
}

ParamsGetter is implemented by any value that has the ParamsList method, which should return the list of parameters of one or more models.

type ParamsSerializer 

type ParamsSerializer struct {
	Model
}

ParamsSerializer allows serialization and deserialization of all parameters of a given Model.

func NewParamsSerializer 

func NewParamsSerializer(m Model) *ParamsSerializer

NewParamsSerializer returns a new ParamsSerializer.

func (*ParamsSerializer) Deserialize 

func (m *ParamsSerializer) Deserialize(r io.Reader) (n int, err error)

Deserialize assigns the params with the values obtained from the reader.

func (*ParamsSerializer) Serialize 

func (m *ParamsSerializer) Serialize(w io.Writer) (n int, err error)

Serialize dumps the params values to the writer.

type ParamsType 

type ParamsType uint8

ParamsType is the enumeration-like type used for the set of parameter (Param) types of a neural network Model. 

const (
	// Weights identifies a Param containing weights.
	Weights ParamsType = iota
	// Biases identifies a Param containing biases.
	Biases
	// Undefined identifies a generic Param, which cannot be described
	// with other ParamsType values.
	Undefined
)

func (ParamsType) String 

func (t ParamsType) String() string

type Payload 

type Payload struct {
	Label int
	Data  []mat.Matrix
}

Payload contains the support data used for example by the optimization methods

func NewEmptySupport 

func NewEmptySupport() *Payload

NewEmptySupport returns an empty support structure, not connected to any optimization method.

func NewPayloadUnmarshalBinaryFrom 

func NewPayloadUnmarshalBinaryFrom(r io.Reader) (*Payload, int, error)

NewPayloadUnmarshalBinaryFrom reads a Payload from the given reader.

type ProcessingMode 

type ProcessingMode int

ProcessingMode regulates the different usage of some operations (e.g. Dropout, BatchNorm, etc.), depending on whether you're doing training or inference. Failing to set the right mode will yield inconsistent inference results. 

const (
	// Training is to be used during the training phase of a model. For example, dropouts are enabled.
	Training ProcessingMode = iota
	// Inference keeps weights fixed while using the model and disables some operations (e.g. skip dropout).
	Inference
)

type StandardForwarder added in v0.2.0 

type StandardForwarder interface {
	// Forward executes the forward step for each input and returns the result.
	// Recurrent networks, treats the input nodes as a sequence. Differently, feed-forward
	// networks are stateless so every computation is independent and possibly concurrent.
	Forward(xs ...ag.Node) []ag.Node
}

StandardForwarder consists of a Forward variadic function that accepts ag.Node and returns a slice of ag.Node. It is called StandardForwarder since this is the most frequent forward method among all implemented neural models.

type StandardModel added in v0.2.0 

type StandardModel interface {
	Model
	StandardForwarder
}

StandardModel consists of a model that implements StandardForwarder.

Source Files

View all Source files

Directories

Path Synopsis
lshattention
Package lshattention provides an implementation of the LSH-Attention model, as describe in `Reformer: The Efficient Transformer` by N. Kitaev, Ł. Kaiser, A. Levskaya (https://arxiv.org/pdf/2001.04451.pdf).
 Package lshattention provides an implementation of the LSH-Attention model, as describe in `Reformer: The Efficient Transformer` by N. Kitaev, Ł. Kaiser, A. Levskaya (https://arxiv.org/pdf/2001.04451.pdf).
multiheadattention
selfattention
syntheticattention
Package syntheticattention provides an implementation of the Synthetic Attention described in: "SYNTHESIZER: Rethinking Self-Attention in Transformer Models" by Tay et al., 2020.
 Package syntheticattention provides an implementation of the Synthetic Attention described in: "SYNTHESIZER: Rethinking Self-Attention in Transformer Models" by Tay et al., 2020.
Package birnncrf provides an implementation of a Bidirectional Recurrent Neural Network (BiRNN) with a Conditional Random Fields (CRF) on tom.
 Package birnncrf provides an implementation of a Bidirectional Recurrent Neural Network (BiRNN) with a Conditional Random Fields (CRF) on tom.
Package bls provides an implementation of the Broad Learning System (BLS) described in "Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture" by C. L. Philip Chen and Zhulin Liu, 2017.
 Package bls provides an implementation of the Broad Learning System (BLS) described in "Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture" by C. L. Philip Chen and Zhulin Liu, 2017.
gnn
slstm
Package slstm implements a Sentence-State LSTM graph neural network.
 Package slstm implements a Sentence-State LSTM graph neural network.
startransformer
Package startransformer provides a variant implementation of the Star-Transformer model introduced by Qipeng Guo, Xipeng Qiu et al.
 Package startransformer provides a variant implementation of the Star-Transformer model introduced by Qipeng Guo, Xipeng Qiu et al.
normalization
adanorm
Package adanorm implements the Adaptive Normalization (AdaNorm) method.
 Package adanorm implements the Adaptive Normalization (AdaNorm) method.
batchnorm
fixnorm
Package fixnorm implements the fixnorm normalization method.
 Package fixnorm implements the fixnorm normalization method.
layernorm
Package layernorm implements the Layer Normalization (LayerNorm) i method.
 Package layernorm implements the Layer Normalization (LayerNorm) i method.
layernormsimple
Package layernormsimple implements a simple version of LayerNorm (LayerNorm-simple).
 Package layernormsimple implements a simple version of LayerNorm (LayerNorm-simple).
rmsnorm
Package rmsnorm implements the Root Mean Square Layer Normalization method.
 Package rmsnorm implements the Root Mean Square Layer Normalization method.
scalenorm
Package rae provides an implementation of the recursive auto-encoder strategy described in "Towards Lossless Encoding of Sentences" by Prato et al., 2019.
 Package rae provides an implementation of the recursive auto-encoder strategy described in "Towards Lossless Encoding of Sentences" by Prato et al., 2019.
Package rc contains built-in Residual Connections (RC).
 Package rc contains built-in Residual Connections (RC).
recurrent
cfn
deltarnn
fsmn
gru
horn
Package horn provides an implementation of Higher Order Recurrent Neural Networks (HORN).
 Package horn provides an implementation of Higher Order Recurrent Neural Networks (HORN).
indrnn
lstm
lstmsc
Package lstmsc provides an implementation of LSTM enriched with a PolicyGradient to enable Dynamic Skip Connections.
 Package lstmsc provides an implementation of LSTM enriched with a PolicyGradient to enable Dynamic Skip Connections.
ltm
mist
Package mist provides an implementation of the MIST (MIxed hiSTory) recurrent network as described in "Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies" by Di Pietro et al., 2018 (https://arxiv.org/pdf/1702.07805.pdf).
 Package mist provides an implementation of the MIST (MIxed hiSTory) recurrent network as described in "Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies" by Di Pietro et al., 2018 (https://arxiv.org/pdf/1702.07805.pdf).
nru
Package nru provides an implementation of the NRU (Non-Saturating Recurrent Units) recurrent network as described in "Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies" by Chandar et al., 2019.
 Package nru provides an implementation of the NRU (Non-Saturating Recurrent Units) recurrent network as described in "Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies" by Chandar et al., 2019.
ran
rla
Package rla provides an implementation of RLA (Recurrent Linear Attention).
 Package rla provides an implementation of RLA (Recurrent Linear Attention).
srn
srnn
Package srnn implements the SRNN (Shuffling Recurrent Neural Networks) by Rotman and Wolf, 2020.
 Package srnn implements the SRNN (Shuffling Recurrent Neural Networks) by Rotman and Wolf, 2020.
tpr

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.