xor

README

xor

XOR code engine in pure Go

more than 200GB/S per core

Introduction:

1. Use SIMD (SSE2 or AVX2 or AVX512) for speeding up

Documentation

See the associated GoDoc

Performance

Performance depends mainly on:

1. SIMD extension
2. non-temporal size
3. hardware ( CPU RAM etc)

Example of performance on my AWS c5d.large (enable AVX512)

speed = ( shards * size ) / cost

data_shards	shard_size	speed (MB/S)
5	4KB	207751.51
5	64KB	98743.27

Documentation

Index

• Variables
• func Encode(dst []byte, src [][]byte) (n int)
• func Replace(parity []byte, data [][]byte) int
• func Update(oldData, newData, parity []byte) int

Variables

var EnableAVX512 = true

AVX512 may slow down CPU Clock (maybe not). TODO need more research: https://lemire.me/blog/2018/04/19/by-how-much-does-avx-512-slow-down-your-cpu-a-first-experiment/

Functions

func Encode

func Encode(dst []byte, src [][]byte) (n int)

Encode encodes elements from source slice into a destination slice. The source and destination may overlap. Encode returns the number of elements encoded, which will be the minimum of len(src[i]) and len(dst).

func Replace

func Replace(parity []byte, data [][]byte) int

Replace replaces oldData vectors with 0 or replaces 0 with newData vectors. Returns the minimum length of data[i] and parity.

It's used in two situations: 1. We didn't have enough data for filling in a stripe, but still did xor encode, we need replace several zero vectors with new vectors which have data after we get enough data finally. 2. After compact, we may have several useless vectors in a stripe, we need replaces these useless vectors with zero vectors for free space.

func Update

func Update(oldData, newData, parity []byte) int

Update updates parity when there is one data vector being changed. Returns the minimum length of oldData and newData and parity. e.g. oldData ⊕ A = parity Pass oldData, newData, parity as args, then the parity will be: parity = newData ⊕ A