rc

Documentation

Overview

Package rc contains built-in Residual Connections (RC).

Index

• func PostNorm(g *ag.Graph, f func(...ag.Node) []ag.Node, norm func(...ag.Node) []ag.Node, ...) []ag.Node
• func PreNorm(g *ag.Graph, f func(...ag.Node) []ag.Node, norm func(...ag.Node) []ag.Node, ...) []ag.Node
• func ReZero(g *ag.Graph, f func(...ag.Node) []ag.Node, alpha ag.Node, xs ...ag.Node) []ag.Node

Functions

func PostNorm

func PostNorm(
	g *ag.Graph,
	f func(...ag.Node) []ag.Node,
	norm func(...ag.Node) []ag.Node,
	xs ...ag.Node,
) []ag.Node

PostNorm performs post-norm residual connections:

y = Norm(x + F(x))

func PreNorm

func PreNorm(
	g *ag.Graph,
	f func(...ag.Node) []ag.Node,
	norm func(...ag.Node) []ag.Node,
	xs ...ag.Node,
) []ag.Node

PreNorm performs pre-norm residual connections:

y = x + F(Norm(x))

func ReZero

func ReZero(
	g *ag.Graph,
	f func(...ag.Node) []ag.Node,
	alpha ag.Node,
	xs ...ag.Node,
) []ag.Node

ReZero performs residual connections by rescaling the function with an alpha learnable parameter (Bachlechner et al., 2020):

y = x + alpha * F(x)