swish

README

Swish

An optimized Swish activation function (Ramachandran, Zoph and Le, 2017), for neural networks.

Screenshots

The graphs above were drawn using the program in cmd/graph, which uses goterm.

Benchmark Results

Using a Swish function that uses math.Exp

First run:

goos: linux
goarch: amd64
pkg: github.com/xyproto/swish
BenchmarkSwish07-8   	200000000	         8.93 ns/op
BenchmarkSwish03-8   	200000000	         8.95 ns/op
PASS
ok  	github.com/xyproto/swish	5.391s

Using the optimized Swish function that uses exp256

goos: linux
goarch: amd64
pkg: github.com/xyproto/swish
BenchmarkSwish07-8   	2000000000	         0.26 ns/op
BenchmarkSwish03-8   	2000000000	         0.26 ns/op
PASS
ok  	github.com/xyproto/swish	1.108s

The optimized Swish function is 34x faster than the one that uses math.Exp, and quite a bit faster than my (apparently bad) attempt at a hand-written assembly version.

The average error (difference in output value) between the optimized and non-optimized version is +-0.0013 and the maximum error is +-0.0024. This is for x in the range [5,3]. See the program in cmd/precision for how this was calculated.

0.00015
0.00001
goos: linux
goarch: amd64
pkg: github.com/xyproto/swish
BenchmarkSwishAssembly07-8      500000000                3.63 ns/op
BenchmarkSwishAssembly03-8      500000000                3.65 ns/op
BenchmarkSwish07-8              2000000000               0.30 ns/op
BenchmarkSwish03-8              2000000000               0.26 ns/op
BenchmarkSwishPrecise07-8       200000000                9.07 ns/op
BenchmarkSwishPrecise03-8       200000000                9.25 ns/op
PASS
ok      github.com/xyproto/swish        11.100s

I have no idea why the assembly version is so slow, but 0.26 ns/op isn't bad for a non-hand-optimized version.

General info

• Version: 1.3.0
• License: MIT
• Author: Alexander F. Rødseth <xyproto@archlinux.org>

Documentation

Index

• func Gaussian01(x float64) float64
• func Gaussian01Precise(x float64) float64
• func Sigmoid(x float64) float64
• func SigmoidPrecise(x float64) float64
• func SoftPlus(x float64) float64
• func SoftPlusPrecise(x float64) float64
• func Swish(x float64) float64
• func SwishAssembly(x float64) float64
• func SwishPrecise(x float64) float64

Functions

func Gaussian01

func Gaussian01(x float64) float64

Gaussian01 is the Gaussian function with mean 0 and sigma 1, using exp256

func Gaussian01Precise

func Gaussian01Precise(x float64) float64

Gaussian01 is the Gaussian function with mean 0 and sigma 1, using math.Exp

func Sigmoid

func Sigmoid(x float64) float64

Sigmoid is the 1 / (1 + exp(-x)) activation function, using exp256

func SigmoidPrecise

func SigmoidPrecise(x float64) float64

SigmoidPrecise is the 1 / (1 + exp(-x)) activation function, using math.Exp

func SoftPlus

func SoftPlus(x float64) float64

SoftPlus is the log(1 + exp(x)) function, using exp256

func SoftPlusPrecise

func SoftPlusPrecise(x float64) float64

SoftPlusPrecise is the log(1 + exp(x)) function, using math.Exp

func Swish

func Swish(x float64) float64

Swish is the x / (1 + exp(-x)) activation function, using exp256

func SwishAssembly

func SwishAssembly(x float64) float64

SwishAssembly is the swish function, written in hand-optimized assembly go: noescape

func SwishPrecise

func SwishPrecise(x float64) float64

SwishPrecise is the x / (1 + exp(-x)) activation function, using math.Exp