compactstream

Documentation

Index

• Constants
• func CaptureTarHeaderBytes(hdr *tar.Header) ([]byte, error)
• func HasMagic(prefix []byte) bool
• func Reconstruct(ctx context.Context, index io.Reader, store BlobStore, output io.Writer) error
• func ReconstructUncompressed(ctx context.Context, index io.Reader, store BlobStore, output io.Writer) error
• type BlobStore
• type CASReference
• type Header
  • func ReadHeader(r io.Reader) (Header, error)
  • func (h Header) RefCount() int
  • func (h Header) RefEntrySize() int
• type Info
  • func Inspect(r io.Reader) (*Info, error)
  • func (i *Info) ReconstructedSize() uint64
  • func (i *Info) ReferencedBytes() uint64
• type NullBlobStore
  • func (NullBlobStore) ReaderForBlob(_ context.Context, _ []byte, size int64) (io.ReadCloser, error)
• type OriginalCompressionInfo
• type ReconstructingReader
  • func NewReconstructingReader(ctx context.Context, index io.Reader, store BlobStore) (*ReconstructingReader, error)
  • func (r *ReconstructingReader) Close() error
  • func (r *ReconstructingReader) Offset() int64
  • func (r *ReconstructingReader) Read(p []byte) (int, error)
  • func (r *ReconstructingReader) RefDigestAt(offset, size int64) ([]byte, bool)
• type Writer
  • func NewWriter(w io.Writer, hashAlgo uint16, hashSize uint16, streamCompression uint8, ...) *Writer
  • func (w *Writer) Close() error
  • func (w *Writer) InlineThreshold() int64
  • func (w *Writer) SetCompressedStreamInfo(digest []byte, size uint64) error
  • func (w *Writer) WriteCASRef(digest []byte, size uint64) error
  • func (w *Writer) WriteStreamBytes(data []byte) error

Constants

const (

	// FormatVersion is the on-disk format version written to and validated in
	// the header.
	FormatVersion uint8 = 0x01

	HashAlgoSHA256 uint16 = 1

	StreamCompressionNone uint8 = 0
	StreamCompressionZstd uint8 = 1

	OriginalCompressionNone uint8 = 0
	OriginalCompressionGzip uint8 = 1
	OriginalCompressionZstd uint8 = 2
)

const MagicSize = len(magic)

MagicSize is the number of leading bytes that identify a compact stream. A caller can peek this many bytes and pass them to HasMagic to detect a compact stream without parsing the full header.

Functions

func CaptureTarHeaderBytes

func CaptureTarHeaderBytes(hdr *tar.Header) ([]byte, error)

func HasMagic

func HasMagic(prefix []byte) bool

HasMagic reports whether prefix begins with the compact stream signature. It only inspects the first MagicSize bytes, so a shorter prefix (e.g. from a truncated read) reports false rather than erroring.

func Reconstruct

func Reconstruct(ctx context.Context, index io.Reader, store BlobStore, output io.Writer) error

func ReconstructUncompressed

func ReconstructUncompressed(ctx context.Context, index io.Reader, store BlobStore, output io.Writer) error

ReconstructUncompressed rebuilds the *uncompressed* layer tar from a compact stream by interleaving the decompressed byte stream with the blobs supplied by store, writing the raw tar bytes to output. Unlike Reconstruct it performs no re-compression and does not validate the compressed-stream digest: the output is the plain tar, not the original compressed file.

Pair it with NullBlobStore to recover only the tar structure (all headers, any inlined small files, and tar block padding) when the content-addressed blobs are unavailable; the omitted blob ranges are then filled with the right number of zero bytes, so a standard archive/tar reader can walk every header (skipping the zeroed bodies) without access to the content store.

Types

type BlobStore

type BlobStore interface {
	ReaderForBlob(ctx context.Context, digest []byte, size int64) (io.ReadCloser, error)
}

type CASReference

type CASReference struct {
	Offset uint64 // byte offset of the range in the reconstructed (uncompressed) stream
	Digest []byte // content digest of the referenced blob
	Size   uint64 // number of bytes the blob contributes to the stream
}

CASReference describes one contiguous range of the reconstructed stream that is stored as a content-addressed blob rather than inline in the byte stream.

type Header

type Header struct {
	Version             uint8
	HashAlgo            uint16
	HashSize            int
	StreamCompression   uint8
	OriginalCompression OriginalCompressionInfo
	RefTableOffset      uint64
	RefTableSize        uint64
	StreamOffset        uint64
	StreamSize          uint64
	// HasCompressedStreamInfo reports whether the optional compressed-stream
	// digest and size fields below are present in the header.
	HasCompressedStreamInfo bool
	// CompressedStreamDigest is the digest of the reconstructed, compressed
	// stream (the original compressed file). Valid only when
	// HasCompressedStreamInfo is true.
	CompressedStreamDigest []byte
	// CompressedStreamSize is the size in bytes of the reconstructed, compressed
	// stream. Valid only when HasCompressedStreamInfo is true.
	CompressedStreamSize uint64
}

Header holds the parsed fields of a compact stream header. See docs/compact-stream.md for the on-disk layout.

func ReadHeader

func ReadHeader(r io.Reader) (Header, error)

ReadHeader reads and validates the fixed-size compact stream header from r. It consumes exactly headerSize bytes, leaving r positioned at the start of the CAS reference table.

func (Header) RefCount

func (h Header) RefCount() int

RefCount is the number of CAS reference entries in the reference table.

func (Header) RefEntrySize

func (h Header) RefEntrySize() int

RefEntrySize is the size in bytes of a single CAS reference table entry (an 8-byte offset, the digest, and an 8-byte size).

type Info

type Info struct {
	Header Header
	Refs   []CASReference
	// StreamUncompressedSize is the size of the byte stream section after
	// decompression: the parts of the reconstructed stream that are NOT replaced
	// by CAS references (tar headers, any inlined small files, and tar block
	// padding).
	StreamUncompressedSize uint64
}

Info is a structural view of a compact stream sufficient to describe and measure it without fetching any CAS blobs (i.e. without reconstruction).

func Inspect

func Inspect(r io.Reader) (*Info, error)

Inspect reads a compact stream in full and returns a structural view of it (header, CAS references, and the decompressed byte-stream length) without fetching any blobs. The byte stream is decompressed only to measure its size; its contents are discarded.

func (*Info) ReconstructedSize

func (i *Info) ReconstructedSize() uint64

ReconstructedSize is the size of the reconstructed, uncompressed stream (the original tar): the inline byte-stream bytes plus all CAS-referenced bytes.

func (*Info) ReferencedBytes

func (i *Info) ReferencedBytes() uint64

ReferencedBytes is the total number of bytes stored as CAS references, i.e. the content held in the content-addressed store rather than in the index.

type NullBlobStore

type NullBlobStore struct{}

NullBlobStore is a BlobStore that returns an equal-length run of NUL bytes for every requested blob, ignoring the digest. It lets ReconstructUncompressed recover the tar structure from a compact stream alone, without the content-addressed store: each omitted blob is replaced by zeros of the same size, yielding a valid tar whose file bodies are zeroed. This is sufficient to read all metadata (headers, link targets, sizes, modes, ...) with archive/tar, but the file contents are not recoverable.

func (NullBlobStore) ReaderForBlob

func (NullBlobStore) ReaderForBlob(_ context.Context, _ []byte, size int64) (io.ReadCloser, error)

type OriginalCompressionInfo

type OriginalCompressionInfo struct {
	Compression      uint8
	Seekable         bool
	CompressionLevel int8
	CompressorJobs   uint8
	EndPadding       uint32
}

type ReconstructingReader

type ReconstructingReader struct {
	// contains filtered or unexported fields
}

ReconstructingReader is an io.Reader over the *uncompressed* layer tar reconstructed from a compact stream: it interleaves the decompressed byte stream (tar headers, inlined small files, and block padding) with the blobs supplied by store at their recorded offsets. Pair it with NullBlobStore to zero-fill the CAS-referenced content when the content store is unavailable, so a standard archive/tar reader can still walk every header.

It additionally tracks the current output offset and can report the digest a compact stream recorded for a file's content, which lets a consumer attach content digests to tar entries without a content store: for a CAS-referenced file, RefDigestAt returns the recorded digest (the sha256 of the file content); for an inlined file the content is present verbatim in the stream and can be hashed by reading it through this reader.

func NewReconstructingReader

func NewReconstructingReader(ctx context.Context, index io.Reader, store BlobStore) (*ReconstructingReader, error)

NewReconstructingReader reads and validates the compact stream header and CAS reference table from index, leaving index positioned at the byte stream, and returns a reader that reconstructs the uncompressed tar on demand.

func (*ReconstructingReader) Close

func (r *ReconstructingReader) Close() error

Close releases the byte-stream decoder and any in-flight CAS blob reader.

func (*ReconstructingReader) Offset

func (r *ReconstructingReader) Offset() int64

Offset returns the number of reconstructed (uncompressed) tar bytes produced so far. After archive/tar's Reader.Next() returns, it equals the byte offset of the current entry's content in the uncompressed tar, which is the key against which RefDigestAt is queried.

func (*ReconstructingReader) Read

func (r *ReconstructingReader) Read(p []byte) (int, error)

func (*ReconstructingReader) RefDigestAt

func (r *ReconstructingReader) RefDigestAt(offset, size int64) ([]byte, bool)

RefDigestAt reports the digest a CAS reference recorded for the content range starting at offset and spanning size bytes, if such a reference exists. The digest is the sha256 of the file content. It returns (nil, false) when the content is not CAS-referenced (e.g. an inlined small file), in which case the caller should hash the content read through this reader instead.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

func NewWriter

func NewWriter(w io.Writer, hashAlgo uint16, hashSize uint16,
	streamCompression uint8, originalCompression OriginalCompressionInfo,
	inlineThreshold int64) *Writer

func (*Writer) Close

func (w *Writer) Close() error

func (*Writer) InlineThreshold

func (w *Writer) InlineThreshold() int64

func (*Writer) SetCompressedStreamInfo

func (w *Writer) SetCompressedStreamInfo(digest []byte, size uint64) error

SetCompressedStreamInfo records the digest and size of the reconstructed, compressed stream (the original compressed file). These are optional: when set, they are written to the header and validated during reconstruction. The digest length must match the index's hash size.

This information is cheap to capture when the file is produced in a single pass (the compressor already computes it) but unknown when an index is built incrementally, hence its optionality.

func (*Writer) WriteCASRef

func (w *Writer) WriteCASRef(digest []byte, size uint64) error

func (*Writer) WriteStreamBytes

func (w *Writer) WriteStreamBytes(data []byte) error