chunkstore

Documentation

Overview

Package chunkstore stores image layers as content-defined chunks instead of one object per layer.

A registry addresses whole layers, so editing one file in a 2 GB layer makes the entire layer a new blob. s3lo is not a registry and is not bound by that: a layer is split into content-defined chunks (see pkg/chunk), each chunk is stored once per bucket under its own digest, and the layer itself becomes an ordered list of chunk digests — a recipe.

Layer digests, the image config, and therefore the image ID are untouched. Only the way the layer's bytes are laid out in the bucket changes, and Fetch verifies the reassembled bytes against the layer digest before returning.

Index

• Constants
• func ChunkKey(chunkDigest string) string
• func ChunksFor(recipe Recipe, entry FileEntry) (need, total int)
• func ExtractFile(ctx context.Context, client storage.Backend, bucket string, recipe Recipe, ...) ([]byte, error)
• func Fetch(ctx context.Context, client storage.Backend, bucket string, recipe Recipe, ...) error
• func IndexKey(compressedDigest string) string
• func NormalisePath(p string) string
• func RecipeKey(layerDigest string) string
• func Store(ctx context.Context, client storage.Backend, ...) (Recipe, Stats, error)
• func StoreIndex(ctx context.Context, client storage.Backend, bucket, compressedDigest string, ...) error
• func StreamCompressed(ctx context.Context, client storage.Backend, bucket string, recipe Recipe, ...) error
• type ChunkRef
• type FileEntry
• type FileIndex
  • func BuildIndex(localPath, layerDigest string) (FileIndex, error)
  • func LoadIndex(ctx context.Context, client storage.Backend, bucket, compressedDigest string) (FileIndex, bool, error)
  • func (ix FileIndex) Find(path string) (FileEntry, bool)
• type Recipe
  • func LoadRecipe(ctx context.Context, client storage.Backend, bucket, layerDigest string) (Recipe, bool, error)
• type Stats
  • func (s Stats) Deduplicated() float64

Constants

const (
	// ChunksPrefix holds chunk objects, shared across every image in the bucket.
	ChunksPrefix = "chunks/sha256/"
	// RecipesPrefix holds one recipe per chunked layer, keyed by layer digest.
	RecipesPrefix = "recipes/sha256/"
)

const IndexesPrefix = "indexes/sha256/"

IndexesPrefix holds one file index per chunked layer, keyed the same way as the recipe.

Functions

func ChunkKey

func ChunkKey(chunkDigest string) string

ChunkKey returns the object key holding a chunk.

func ChunksFor

func ChunksFor(recipe Recipe, entry FileEntry) (need, total int)

ChunksFor reports how many chunks a read of entry has to fetch, and how many the whole layer holds. It is what lets a caller say what the index saved.

func ExtractFile

func ExtractFile(ctx context.Context, client storage.Backend, bucket string, recipe Recipe, entry FileEntry) ([]byte, error)

ExtractFile returns one file's bytes, fetching only the chunks its bytes fall in. This is the whole point of the index: a 4 MB read out of a 2 GB layer moves one or two chunks, not the layer.

func Fetch

func Fetch(ctx context.Context, client storage.Backend, bucket string, recipe Recipe, destPath string) error

Fetch reassembles a chunked layer into destPath and verifies that the result hashes to recipe.Layer. Chunks are fetched concurrently and written at their own offsets, so peak memory is bounded by chunkConcurrency, not layer size.

func IndexKey

func IndexKey(compressedDigest string) string

IndexKey returns the object key holding the file index for a layer.

func NormalisePath

func NormalisePath(p string) string

NormalisePath strips the leading forms a tar entry may carry so two spellings of one path compare equal.

func RecipeKey

func RecipeKey(layerDigest string) string

RecipeKey returns the object key holding the recipe for a layer digest.

func Store

func Store(ctx context.Context, client storage.Backend, bucket, localPath, layerDigest string) (Recipe, Stats, error)

Store splits localPath into chunks, uploads the ones the bucket does not already have, and writes the recipe. layerDigest is the hex sha256 of localPath's raw contents. The returned Recipe carries the compressed identity the image manifest must reference.

Compression happens on the splitting goroutine rather than in the uploaders, because the compressed digest is a hash over the frames in order and cannot be assembled out of order. Uploads still run concurrently.

func StoreIndex

func StoreIndex(ctx context.Context, client storage.Backend, bucket, compressedDigest string, ix FileIndex) error

StoreIndex writes the index for a layer, zstd-compressed. A layer with 20k files produces a couple of megabytes of JSON, and this object is fetched on every per-file read.

func StreamCompressed

func StreamCompressed(ctx context.Context, client storage.Backend, bucket string, recipe Recipe, w io.Writer) error

StreamCompressed writes the layer's chunk objects to w exactly as stored, in order, without decompressing anything.

The result is a valid zstd stream whose digest is recipe.CompressedDigest, which is what the image manifest advertises. This is the fast read path: it moves roughly a third of the bytes the raw form does, and the decompression happens in the client that was always going to decompress a layer anyway.

Types

type ChunkRef

type ChunkRef struct {
	Digest         string `json:"digest"`
	Size           int64  `json:"size"`
	CompressedSize int64  `json:"compressedSize"`
}

ChunkRef identifies one chunk. Size is its length in the assembled raw layer; CompressedSize is the size of the stored object.

type FileEntry

type FileEntry struct {
	Path   string `json:"path"`
	Offset int64  `json:"offset,omitempty"`
	Size   int64  `json:"size,omitempty"`
	Link   string `json:"link,omitempty"`
}

FileEntry is one entry inside the layer. Whiteout markers are ordinary zero-length entries and are recorded like any other, so a reader can see that a higher layer deleted a path without fetching anything.

Link is set for a symlink or hard link, in which case Offset and Size mean nothing. Links are indexed because leaving them out makes the index lie about the image: /etc/os-release is a symlink on every Debian-derived image, and a reader that skipped links would report it missing.

type FileIndex

type FileIndex struct {
	// Layer is the raw layer digest the offsets belong to.
	Layer string      `json:"layer"`
	Files []FileEntry `json:"files"`
}

FileIndex records where each file's bytes sit inside a layer's raw tar.

It deliberately does not record which chunks hold a file. Offsets plus the recipe's chunk sizes give that answer exactly, and storing it twice would only create a second thing that can disagree with the first. Slacker (USENIX FAST '16) measured that pulling is 76% of container start time while 6.4% of the image is ever read; addressing a file rather than a layer is what this index makes possible.

func BuildIndex

func BuildIndex(localPath, layerDigest string) (FileIndex, error)

BuildIndex walks the raw tar at localPath and records where each file's data begins. A layer that is not a tar yields no index and no error: chunking has to keep working for whatever bytes it is handed.

func LoadIndex

func LoadIndex(ctx context.Context, client storage.Backend, bucket, compressedDigest string) (FileIndex, bool, error)

LoadIndex reads the index for a layer. It returns false when there is none, which is the normal case for a layer stored whole or pushed before indexes existed.

func (FileIndex) Find

func (ix FileIndex) Find(path string) (FileEntry, bool)

Find returns the entry for path, or false. Both sides are normalised, because a layer tar may name the same file "usr/bin/tool", "./usr/bin/tool" or "/usr/bin/tool" depending on what built it.

type Recipe

type Recipe struct {
	// Layer is the hex sha256 of the assembled raw layer, without "sha256:".
	Layer string `json:"layer"`
	Size  int64  `json:"size"`
	// CompressedDigest is the hex sha256 of the chunk objects concatenated in
	// order, and is the key this recipe is stored under.
	CompressedDigest string     `json:"compressedDigest"`
	CompressedSize   int64      `json:"compressedSize"`
	Chunks           []ChunkRef `json:"chunks"`
}

Recipe is the ordered chunk list that reconstitutes one layer, in both the forms the layer can take.

Layer/Size describe the raw tar: that digest is the image config's diff_id and never changes. CompressedDigest/CompressedSize describe the concatenation of the stored chunk objects, which is itself a valid zstd stream because zstd frames concatenate. That second identity is what the image manifest references, so a client can be handed the compressed bytes and decompress them natively instead of having s3lo decompress on its behalf.

func LoadRecipe

func LoadRecipe(ctx context.Context, client storage.Backend, bucket, layerDigest string) (Recipe, bool, error)

LoadRecipe reads the recipe stored under digest. The key is the layer's compressed digest, which is exactly what an image manifest references, so callers can pass a manifest layer digest straight through.

It returns false when there is no recipe, so a caller can fall back to a whole-layer blob without treating a plain bucket as an error.

type Stats

type Stats struct {
	Chunks         int
	ChunksUploaded int
	Bytes          int64
	BytesUploaded  int64
}

Stats reports what a Store call actually transferred, which is the whole point of chunking: Uploaded is the part that was not already in the bucket.

func (Stats) Deduplicated

func (s Stats) Deduplicated() float64

Deduplicated returns the fraction of bytes that did not need uploading.