adanorm

Documentation

Overview

Reference: "Understanding and Improving Layer Normalization" by Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, Junyang Lin (2019). (https://papers.nips.cc/paper/8689-understanding-and-improving-layer-normalization.pdf)

Index

• type Model
  • func New(scale float64) *Model
  • func (m *Model) NewProc(ctx nn.Context) nn.Processor
• type Processor
  • func (p *Processor) Forward(xs ...ag.Node) []ag.Node
  • func (p *Processor) Mean(xs []ag.Node) []ag.Node
  • func (p *Processor) StdDev(meanVectors []ag.Node, xs []ag.Node) []ag.Node

Types

type Model

type Model struct {
	// contains filtered or unexported fields
}

Model contains the scaling factor.

func New

func New(scale float64) *Model

New returns a new model.

func (*Model) NewProc

func (m *Model) NewProc(ctx nn.Context) nn.Processor

NewProc returns a new processor to execute the forward step.

type Processor

type Processor struct {
	nn.BaseProcessor
	// contains filtered or unexported fields
}

func (*Processor) Forward

func (p *Processor) Forward(xs ...ag.Node) []ag.Node

Forward performs the forward step for each input and returns the result.

func (*Processor) Mean

func (p *Processor) Mean(xs []ag.Node) []ag.Node

func (*Processor) StdDev

func (p *Processor) StdDev(meanVectors []ag.Node, xs []ag.Node) []ag.Node