filechecksum

Documentation

Overview

package filechecksum provides the FileChecksumGenerator, whose main responsibility is to read a file, and generate both weak and strong checksums for every block. It is also used by the comparer, which will generate weak checksums for potential byte ranges that could match the index, and strong checksums if needed.

Index

• Variables
• type ChecksumLookup
• type ChecksumResults
• type CompressionFunction
• type FileChecksumGenerator
  • func NewFileChecksumGenerator(blocksize uint) *FileChecksumGenerator
  • func (check *FileChecksumGenerator) ChecksumSize() int
  • func (check *FileChecksumGenerator) GenerateChecksums(inputFile io.Reader, output io.Writer) (fileChecksum []byte, err error)
  • func (check *FileChecksumGenerator) GetChecksumSizes() (int, int)
  • func (check *FileChecksumGenerator) GetFileHash() hash.Hash
  • func (check *FileChecksumGenerator) GetStrongHash() hash.Hash
  • func (check *FileChecksumGenerator) Reset()
  • func (check *FileChecksumGenerator) StartChecksumGeneration(inputFile io.Reader, blocksPerResult uint, ...) <-chan ChecksumResults
• type HashVerifier
  • func (v *HashVerifier) VerifyBlockRange(startBlockID uint, data []byte) bool
• type RollingHash

Variables

var DefaultFileHashGenerator = func() hash.Hash {
	return md5.New()
}

We provide an overall hash of individual files

var DefaultStrongHashGenerator = func() hash.Hash {
	return md5.New()
}

Rsync swapped to this after version 30 this is a factory function, because we don't actually want to share hash state

Types

type ChecksumLookup

type ChecksumLookup interface {
	GetStrongChecksumForBlock(blockID int) []byte
}

type ChecksumResults

type ChecksumResults struct {
	// Return multiple chunks at once for performance
	Checksums []chunks.ChunkChecksum
	// only used for the last item
	Filechecksum []byte
	// signals that this is the last item
	Err error
}

type CompressionFunction

type CompressionFunction func([]byte) (compressedSize int64, err error)

A function or object that can compress blocks the compression function must also write out the compressed blocks somewhere! Compressed blocks should be independently inflatable

type FileChecksumGenerator

type FileChecksumGenerator struct {
	// See BlockBuffer
	WeakRollingHash  RollingHash
	StrongHash       hash.Hash
	FileChecksumHash hash.Hash
	BlockSize        uint
}

FileChecksumGenerator provides a description of what hashing functions to use to evaluate a file. Since the hashes store state, it is NOT safe to use a generator concurrently for different things.

func NewFileChecksumGenerator

func NewFileChecksumGenerator(blocksize uint) *FileChecksumGenerator

Uses all default hashes (MD5 & rollsum16)

func (*FileChecksumGenerator) ChecksumSize

func (check *FileChecksumGenerator) ChecksumSize() int

func (*FileChecksumGenerator) GenerateChecksums

func (check *FileChecksumGenerator) GenerateChecksums(inputFile io.Reader, output io.Writer) (fileChecksum []byte, err error)

GenerateChecksums reads each block of the input file, and outputs first the weak, then the strong checksum to the output writer. It will return a checksum for the whole file. Potentially speaking, this might be better producing a channel of blocks, which would remove the need for io from a number of other places.

func (*FileChecksumGenerator) GetChecksumSizes

func (check *FileChecksumGenerator) GetChecksumSizes() (int, int)

func (*FileChecksumGenerator) GetFileHash

func (check *FileChecksumGenerator) GetFileHash() hash.Hash

Gets the Hash function for the overall file used on each block defaults to md5

func (*FileChecksumGenerator) GetStrongHash

func (check *FileChecksumGenerator) GetStrongHash() hash.Hash

Gets the Hash function for the strong hash used on each block defaults to md5, but can be overriden by the generator

func (*FileChecksumGenerator) Reset

func (check *FileChecksumGenerator) Reset()

Reset all hashes to initial state

func (*FileChecksumGenerator) StartChecksumGeneration

func (check *FileChecksumGenerator) StartChecksumGeneration(
	inputFile io.Reader,
	blocksPerResult uint,
	compressionFunction CompressionFunction,
) <-chan ChecksumResults

type HashVerifier

type HashVerifier struct {
	BlockSize           uint
	Hash                hash.Hash
	BlockChecksumGetter ChecksumLookup
}

func (*HashVerifier) VerifyBlockRange

func (v *HashVerifier) VerifyBlockRange(startBlockID uint, data []byte) bool

type RollingHash

type RollingHash interface {
	// the size of the hash output
	Size() int

	AddByte(b byte)
	RemoveByte(b byte, length int)

	AddBytes(bs []byte)
	RemoveBytes(bs []byte, length int)

	// pairs up bytes to do remove/add in the right order
	AddAndRemoveBytes(add []byte, remove []byte, length int)

	SetBlock(block []byte)

	GetSum(b []byte)
	Reset()
}