pack

Documentation

Overview

Package pack implements the doomsday pack file format.

Pack format:

[EncryptedBlob1][EncryptedBlob2]...[EncryptedBlobN][EncryptedHeader][HeaderLength:4 bytes LE]

The header is at the END for streaming writes (no seek required — essential for S3/B2). Each blob is independently encrypted with a per-blob derived key. HeaderLength is uint32 little-endian, giving the encrypted header size.

Index

• Constants
• func MarshalHeader(h Header) ([]byte, error)
• func ReadBlob(r io.ReaderAt, entry HeaderEntry) ([]byte, error)
• type DecryptFunc
• type EncryptFunc
• type Header
  • func ReadHeader(r io.ReaderAt, totalSize int64, decrypt DecryptFunc) (Header, error)
  • func UnmarshalHeader(data []byte) (Header, error)
• type HeaderEntry
• type Writer
  • func NewWriter(w io.Writer) *Writer
  • func (pw *Writer) AddBlob(id types.BlobID, blobType types.BlobType, uncompressedLen uint32, ...) error
  • func (pw *Writer) Count() int
  • func (pw *Writer) Entries() Header
  • func (pw *Writer) Finalize(encrypt EncryptFunc) error
  • func (pw *Writer) Size() uint32

Constants

const (
	// MaxHeaderSize is the maximum allowed encrypted header size (64 MiB).
	// A legitimate pack header with 100k entries is a few MB at most.
	MaxHeaderSize = 64 << 20

	// MinHeaderSize is the minimum encrypted header (AES-GCM nonce + tag = 28 bytes).
	MinHeaderSize = 28

	// MaxEntriesPerPack limits the number of entries in a single pack header
	// to prevent OOM from a malicious header with millions of entries.
	MaxEntriesPerPack = 100_000

	// MaxBlobSize is the maximum allowed individual blob size (32 MiB).
	// Chunker output is typically much smaller.
	MaxBlobSize = 32 << 20
)

Functions

func MarshalHeader

func MarshalHeader(h Header) ([]byte, error)

MarshalHeader serializes a header to JSON.

func ReadBlob

func ReadBlob(r io.ReaderAt, entry HeaderEntry) ([]byte, error)

ReadBlob reads a single blob from a pack file using the header entry's offset and length.

Types

type DecryptFunc

type DecryptFunc func(ciphertext []byte) ([]byte, error)

DecryptFunc decrypts a header blob.

type EncryptFunc

type EncryptFunc func(plaintext []byte) ([]byte, error)

EncryptFunc encrypts a header blob. Used to encrypt the pack header.

type Header

type Header []HeaderEntry

Header is the collection of blob entries in a pack file.

func ReadHeader

func ReadHeader(r io.ReaderAt, totalSize int64, decrypt DecryptFunc) (Header, error)

ReadHeader reads and decrypts the header from a pack file. The pack file must be seekable (io.ReaderAt) since the header is at the end.

func UnmarshalHeader

func UnmarshalHeader(data []byte) (Header, error)

UnmarshalHeader deserializes a header from JSON.

type HeaderEntry

type HeaderEntry struct {
	ID                 types.BlobID   `json:"id"`
	Type               types.BlobType `json:"type"`
	Offset             uint32         `json:"offset"`
	Length             uint32         `json:"length"`              // encrypted size in pack
	UncompressedLength uint32         `json:"uncompressed_length"` // 0 if not compressed
}

HeaderEntry describes a single blob within a pack file.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes blobs to a pack file in streaming fashion. The header is written at the end, so no seeking is required.

func NewWriter

func NewWriter(w io.Writer) *Writer

NewWriter creates a pack writer that writes to w.

func (*Writer) AddBlob

func (pw *Writer) AddBlob(id types.BlobID, blobType types.BlobType, uncompressedLen uint32, ciphertext []byte) error

AddBlob writes an encrypted blob to the pack and records it in the header. The caller is responsible for encrypting the blob data before calling this.

func (*Writer) Count

func (pw *Writer) Count() int

Count returns the number of blobs written so far.

func (*Writer) Entries

func (pw *Writer) Entries() Header

Entries returns the current header entries (for testing/inspection).

func (*Writer) Finalize

func (pw *Writer) Finalize(encrypt EncryptFunc) error

Finalize writes the encrypted header and header length to complete the pack file. The encrypt function is used to encrypt the header data.

func (*Writer) Size

func (pw *Writer) Size() uint32

Size returns the current byte offset (total blob data written).