syntheticattention

Documentation

Overview

This is an implementation of the Synthetic Attention described in: "SYNTHESIZER: Rethinking Self-Attention in Transformer Models" by Tay et al., 2020. (https://arxiv.org/pdf/2005.00743.pdf)

Index

• type Config
• type ContextProb
• type Model
  • func New(config Config) *Model
  • func (m *Model) NewProc(ctx nn.Context) nn.Processor
• type Processor
  • func (p *Processor) Forward(xs ...ag.Node) []ag.Node

Types

type Config

type Config struct {
	InputSize  int
	HiddenSize int
	ValueSize  int
	MaxLength  int
}

type ContextProb

type ContextProb struct {
	// contains filtered or unexported fields
}

type Model

type Model struct {
	Config
	FFN   *stack.Model
	Value *linear.Model
	W     *nn.Param `type:"weights"`
}

Model contains the serializable parameters.

func New

func New(config Config) *Model

New returns a new model with parameters initialized to zeros.

func (*Model) NewProc

func (m *Model) NewProc(ctx nn.Context) nn.Processor

NewProc returns a new processor to execute the forward step.

type Processor

type Processor struct {
	nn.BaseProcessor

	Attention *ContextProb
	// contains filtered or unexported fields
}

func (*Processor) Forward

func (p *Processor) Forward(xs ...ag.Node) []ag.Node

Forward performs the forward step for each input and returns the result.