fastcdc

README

go-fastcdc

go-fastcdc is a Golang package implementing the fastCDC chunking algorithm.

    chunkerOpts := fastcdc.NewChunkerOpts()
    chunkerOpts.NormalSize = 8 * 1024,   // 8KB

    chunker, err := fastcdc.NewChunker(rd, chunkerOpts)
    if err != nil {
        log.Fatal(err)
    }

    for {
        chunk, err := chunker.Next()
        if err != nil {
            if err == io.EOF {
                // no more chunks to read
                break
            }
            log.Fatal(err)
        }

        fmt.Println(chunk.Offset, chunk.Size)
        // data is in chunk.Data
    }

Reference

• Xia, Wen, et al. "Fastcdc: a fast and efficient content-defined chunking approach for data deduplication." 2016 USENIX Annual Technical Conference

Documentation

Index

• Variables
• type Chunker
  • func NewChunker(reader io.Reader, options *ChunkerOpts) (*Chunker, error)
  • func (chunker *Chunker) Next() ([]byte, error)
• type ChunkerOpts
  • func NewChunkerOptions() *ChunkerOpts

Variables

var G [256]uint64 = [256]uint64{}/* 256 elements not displayed */

randomly generated Gear table

Types

type Chunker

type Chunker struct {
	NormalSize uint32
	MinSize    uint32
	MaxSize    uint32
	// contains filtered or unexported fields
}

func NewChunker

func NewChunker(reader io.Reader, options *ChunkerOpts) (*Chunker, error)

func (*Chunker) Next

func (chunker *Chunker) Next() ([]byte, error)

type ChunkerOpts

type ChunkerOpts struct {
	NormalSize uint32
	MinSize    uint32
	MaxSize    uint32
}

func NewChunkerOptions

func NewChunkerOptions() *ChunkerOpts