reedlib

README

reedlib

A high-level Reed Solomon erasure encoding library and set of command-line tools written in Go.

Quick Start

Installing the library

Either run:

go get github.com/prologic/reedlib

Or just import the library in your project:

import "github.com/prologic/reedlib"

NOTE: There isn't much of a library yet here at the time of writing this (2020-12-27). Coming soon...

Installing the command-line tools

Install reed-encode:

go get github.com/prologic/reedlib/cmd/reed-encode/....

Install reed-decode:

go get github.com/prologic/reedlib/cmd/reed-decode/....

Usage

Encoding a file (command-line)

To encode a file using Reed-Solomon erasure encoding using the default Data and Parity Shards (3 + 1 respectively):

reed-encode ./testdata/IMG_7895.JPG

This will result in a number of output files from the result of splitting up and encoding the original input file using the specified number of data and parity shards:

$ ls -lah ./testdata/
total 6.4M
drwxr-xr-x  8 prologic staff  256 Dec 27 09:10 .
drwxr-xr-x 17 prologic staff  544 Dec 27 09:12 ..
-rw-r--r--  1 prologic staff 6.1K Dec 27 09:10 .DS_Store
-rw-r--r--  1 prologic staff 2.7M Dec 27 09:10 IMG_7895.JPG
-rw-r--r--  1 prologic staff 916K Dec 27 09:10 IMG_7895.JPG.0
-rw-r--r--  1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.1
-rw-r--r--  1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.2
-rw-r--r--  1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.3

Keep the pieces (data and parity shards) on different storage devices or storage nodes which can then later be used to reconstruct the original file, even if one of the shards is lost or corrupt (parity of one).

Decoding shards (command-line)

To decode a number of shards from a previous encoding using the default Data and Parity Shards (3 + 1 respectively):

First remove the original input file to demonstrate recovery:

rm -f ./testdata/IMG_7895.JPG

Now reconstruct the original input file using the shards:

reed-decode ./testdata/IMG_7895.JPG

You should now have the original file recovered and intact:

$ ls -lah ./testdata/
total 6.3M
drwxr-xr-x  7 prologic staff  224 Dec 27 09:33 .
drwxr-xr-x 15 prologic staff  480 Dec 27 09:33 ..
-rw-r--r--  1 prologic staff 2.7M Dec 27 09:33 IMG_7895.JPG
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.0
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.1
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.2
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.3

You can even remove the original input file and either remove or corrupt one of the shards and the original input file is still recoverable from the remaining shards.

Notes

This is a high-level wrapper library and set of command-line tools that uses @klauspost's reedsolomon library which itself is a Go port of the JavaReedSolomon library released by Backblaze, with some additional optimizations.

For an introduction on erasure coding, see the post on the Backblaze blog.

Some interesting properties to note:

• The number of data + parity shards MUST NOT exceed 256 as the default bit-field size is 8 for Reed Solomon erasure codes.
• You can detect errors or recover from errors from up to parity shards. For example, with a data/parity of 3+1 you can recover from 1 lost or corrupt shard. With a data/parity of 3+2 you can recover from up to two shard failures.

License

reedlib is licensed under the terms of the MIT License

Documentation

Index

• Variables
• func FullVersion() string

Variables

var (
	// Version release version
	Version = "0.0.1"

	// Commit will be overwritten automatically by the build system
	Commit = "HEAD"
)

Functions

func FullVersion

func FullVersion() string

FullVersion display the full version and build