sgu

Documentation

Overview

Package sgu implements the Spatial Gating Unit (SGU). Reference: `Pay Attention to MLPs` by Liu et al, 2021 (https://arxiv.org/pdf/2105.08050.pdf)

Index

• type Config
• type Model
  • func New(config Config) *Model
  • func (m *Model) Forward(xs ...ag.Node) []ag.Node
  • func (m *Model) Initialize(seed uint64)

Types

type Config

type Config struct {
	Dim        int
	DimSeq     int
	InitEps    mat.Float
	Activation ag.OpName
}

Config provides configuration parameters for Model.

type Model

type Model struct {
	nn.BaseModel
	Config Config
	Norm   *layernorm.Model
	Proj   *conv1x1.Model
	Act    *activation.Model
}

Model contains the serializable parameters.

func New

func New(config Config) *Model

New returns a new Model initialized to zeros.

func (*Model) Forward

func (m *Model) Forward(xs ...ag.Node) []ag.Node

Forward performs the forward step for each input node and returns the result.

func (*Model) Initialize

func (m *Model) Initialize(seed uint64)

Initialize set the projection weights as near-zero values and the biases as ones to improve training stability.