startransformer

Documentation

Overview

StarTransformer is a variant of the model introduced by Qipeng Guo, Xipeng Qiu et al. in "Star-Transformer", 2019 (https://www.aclweb.org/anthology/N19-1133.pdf). In this implementation, the Scaled Dot Product Attention is replaced by a Linear Attention.

Index

• type Config
• type Model
  • func New(config Config) *Model
  • func (m *Model) NewProc(ctx nn.Context) nn.Processor
• type Processor
  • func (p *Processor) Forward(xs ...ag.Node) []ag.Node

Types

type Config

type Config struct {
	InputSize int
	QuerySize int
	KeySize   int
	ValueSize int
	Steps     int
}

type Model

type Model struct {
	Config
	Query         *linear.Model
	Key           *linear.Model
	Value         *linear.Model
	RelayQuery    *linear.Model
	RelayKey      *linear.Model
	RelayValue    *linear.Model
	SatelliteNorm *layernorm.Model
	RelayNorm     *layernorm.Model
}

Model contains the serializable parameters.

func New

func New(config Config) *Model

New returns a new model with parameters initialized to zeros.

func (*Model) NewProc

func (m *Model) NewProc(ctx nn.Context) nn.Processor

NewProc returns a new processor to execute the forward step.

type Processor

type Processor struct {
	nn.BaseProcessor

	Steps int
	// contains filtered or unexported fields
}

func (*Processor) Forward

func (p *Processor) Forward(xs ...ag.Node) []ag.Node

Forward performs the forward step returns the results.