chunk

Documentation

Overview

Package chunk implements the versioned content-defined chunking (CDC) that FXVCS uses to store large assets as reusable pieces.

The encoding identifier written into pointers and object manifests names one exact parameter set, never "whatever this binary happens to prefer":

whole/1        the object is one piece: the whole file
cdc-fastcdc/1  FastCDC with normalization level 2, min 64 KiB,
               average 256 KiB, max 1 MiB, and the gear table pinned by
               TestGearTableIsPinned

Those numbers are the winning candidate of the Stage 2 corpus measurement (docs/stage-2-measurement.md), not a guess.

That matters because the chunk list determines the object manifest, the manifest digest is inside the pointer, and the pointer bytes are what Git stores. Two clients that split the same file differently would produce different pointers for identical content and Git would report a spurious modification. Changing any parameter therefore requires a new identifier (cdc-fastcdc/2), never an edit here.

Splitting is streaming and bounded: peak memory is one Params.Max window regardless of file size.

Index

• Constants
• Variables
• func Split(r io.Reader, p Params, fn func(Chunk) error) error
• func SplitFixed(r io.Reader, size int, fn func(Chunk) error) error
• type Chunk
• type Params
  • func ParamsFor(encoding string) (Params, error)
  • func (p Params) Validate() error
• type Policy
  • func DefaultPolicy() Policy
  • func PolicyFromEnv() (Policy, error)
  • func (p Policy) AtLeastMinChunked() int64
  • func (p Policy) Encoding(repoPath string, size int64) string

Constants

const (
	EncodingWhole     = "whole/1"
	EncodingFastCDCv1 = "cdc-fastcdc/1"
)

Encoding identifiers. These are the values that appear in pointers and object manifests.

const EnvPolicy = "FXVCS_CHUNKING"

EnvPolicy is a diagnostic override for the measurement harness and tests. It is documented as diagnostic-only: a machine that sets it produces different (still valid, still verifiable) pointers than its collaborators, so Git will report the affected paths as modified when they are restaged.

FXVCS_CHUNKING=off          never chunk
FXVCS_CHUNKING=always       chunk every object
FXVCS_CHUNKING=min=<bytes>  change only the size floor

Variables

var ErrInvalidParams = errors.New("chunk: invalid parameters")

ErrInvalidParams reports a parameter set that cannot be used.

var FastCDCv1 = Params{
	Encoding:      EncodingFastCDCv1,
	Min:           64 << 10,
	Avg:           256 << 10,
	Max:           1 << 20,
	Normalization: 2,
}

FastCDCv1 is the parameter set named by EncodingFastCDCv1. It is a constant of the format, not a tunable: see the package comment.

Functions

func Split

func Split(r io.Reader, p Params, fn func(Chunk) error) error

Split reads r to EOF and calls fn once per chunk, in order. Peak memory is p.Max bytes plus the reader's own buffering, whatever the input size. An error from fn stops the walk and is returned unchanged.

An empty input produces exactly one empty chunk, so that every object has at least one chunk and the manifest schema's minItems holds.

func SplitFixed

func SplitFixed(r io.Reader, size int, fn func(Chunk) error) error

SplitFixed is the fixed-size splitter used as the measurement baseline in docs/stage-2-measurement.md. It is never used for stored objects: no encoding identifier names it.

Types

type Chunk

type Chunk struct {
	// Offset is the byte offset of the chunk in the input stream.
	Offset int64
	// Data is the chunk content. It aliases the splitter's window and is only
	// valid until the callback returns; copy it if it must outlive the call.
	Data []byte
}

Chunk is one emitted piece.

type Params

type Params struct {
	// Encoding is the identifier written into pointers and manifests.
	Encoding string
	// Min is the smallest chunk the splitter will emit (except the last).
	Min int
	// Avg is the target average chunk size; it must be a power of two.
	Avg int
	// Max is the largest chunk the splitter will emit.
	Max int
	// Normalization is the FastCDC normalized-chunking level in bits: the
	// cut mask is tightened by this many bits below Avg and loosened by the
	// same amount above it, which concentrates chunk sizes around Avg.
	Normalization uint
}

Params is one pinned, versioned parameter set.

func ParamsFor

func ParamsFor(encoding string) (Params, error)

ParamsFor returns the parameter set behind an encoding identifier.

func (Params) Validate

func (p Params) Validate() error

Validate checks the invariants the splitter relies on.

type Policy

type Policy struct {
	// MinChunkedSize is the size at or above which an object is chunked.
	// Below it a single whole object costs one round trip and no manifest.
	MinChunkedSize int64
	// WholeOnly lists lowercase file extensions (with the dot) that stay
	// whole-blob regardless of size because the corpus measurement showed no
	// worthwhile reuse between versions.
	WholeOnly map[string]bool
}

Policy decides which encoding an object gets. It is deliberately a property of the *format generation*, not of a machine: the encoding choice feeds the object manifest, the manifest digest feeds the pointer, and the pointer is what Git stores. Two clients that disagree about the policy would produce different pointers for byte-identical content and Git would report a spurious modification on every restage.

Changing these defaults therefore means introducing a new encoding identifier and a new minimumFXVCSVersion, exactly like changing the FastCDC parameters.

func DefaultPolicy

func DefaultPolicy() Policy

DefaultPolicy is the Stage 2 policy justified by docs/stage-2-measurement.md.

The measurement compared whole blobs, fixed chunks, and four CDC parameter sets over version pairs of RSC7-backed and audio assets. Every format measured cleared the 25% net-transfer gate at cdc-fastcdc/1 parameters, so WholeOnly is empty; the mechanism stays because the report must be re-run against a real asset corpus and a format that falls below the gate there is added here rather than handled with new code.

Objects below MinChunkedSize stay whole: at 1 MiB the manifest, the extra existence checks, and the extra requests cost more than the reuse.

func PolicyFromEnv

func PolicyFromEnv() (Policy, error)

PolicyFromEnv applies EnvPolicy to the default policy.

func (Policy) AtLeastMinChunked

func (p Policy) AtLeastMinChunked() int64

AtLeastMinChunked is the lookahead the size decision needs: the number of leading bytes that must be present before an object counts as large enough to chunk.

func (Policy) Encoding

func (p Policy) Encoding(repoPath string, size int64) string

Encoding returns the encoding identifier for an object. repoPath is the repository-relative slash path (empty when unknown) and size is the exact object size.

Callers must know the size, even when ingesting from a stream: deciding "chunk it" for a file whose size was merely unknown would give the same bytes two different pointers depending on how they arrived. The ingest path therefore reads MinChunkedSize bytes ahead before choosing (see AtLeastMinChunked).