amberpack

Documentation

Overview

Package amberpack defines the Amber-Store pack format. The content-addressed record codec (record.go) is shared by packstore's on-disk segments and the remote-sync wire packs defined here.

A wire pack is a possibly-partial, unordered set of CAS objects (like a git pack) carrying no root key. Layout:

Magic    "AMBERPK\x03"   8 bytes  (plaintext)
Records  repeat: one EncodeRecord output each — a 46-byte header
         (tag 0x01 + key[32] + flags + ulen + slen + CRC) followed by the payload
End      0x00

Each record is the same self-describing, CRC-protected, per-record-zstd unit packstore writes on disk (see record.go); a wire pack is just those records framed by a magic and an explicit end marker, so a truncated stream is detected rather than read as a clean EOF. The Reader validates framing, CRC, and key canonicality and decodes each payload; it does NOT verify the payload hash — that happens in the storage path (packstore WriteParallel with Verify).

Versions 1 and 2 ("AMBERPK\x01" / "AMBERPK\x02") were the older uncompressed and whole-stream-zstd stream formats; they are no longer produced and are rejected by the Reader.

Index

• Constants
• Variables
• func DecodePayload(flags byte, ulen uint32, stored []byte) ([]byte, error)
• func EncodeRecord(k key.Key, data []byte) ([]byte, error)
• type Reader
  • func NewReader(r io.Reader) *Reader
  • func (r *Reader) All() iter.Seq2[fstree.Object, error]
• type Record
  • func ParseRecord(b []byte) (Record, error)
• type Writer
  • func NewWriter(w io.Writer) *Writer
  • func (w *Writer) Add(o fstree.Object) error
  • func (w *Writer) AddRecord(rec []byte) error
  • func (w *Writer) Close() error

Constants

const (
	// RecHeaderSize is the fixed record-header length:
	// tag(1) + key(32) + flags(1) + ulen(4) + slen(4) + crc(4). Payload follows.
	RecHeaderSize = 46
)

Variables

var ErrCorrupt = errors.New("amberpack: corrupt pack data")

ErrCorrupt wraps every record-level corruption error surfaced by ParseRecord and DecodePayload (bad framing, bad flags, CRC mismatch, length inconsistency, non-canonical key). It is the record-level counterpart to the stream-level ErrMalformed; distinguish either with errors.Is. The packstore package aliases this sentinel for its footer- and scrub-level corruption too, so the message stays deliberately general rather than naming "record".

var ErrMalformed = errors.New("amberpack: malformed pack stream")

ErrMalformed wraps every error from a structurally invalid wire pack (bad or legacy magic, truncation, an oversized or bad record, or a corrupt record). Callers distinguish it with errors.Is to map to a client error.

Functions

func DecodePayload

func DecodePayload(flags byte, ulen uint32, stored []byte) ([]byte, error)

DecodePayload returns caller-owned payload bytes from a record's stored payload. stored may be a read-only mmap slice and is never retained.

func EncodeRecord

func EncodeRecord(k key.Key, data []byte) ([]byte, error)

EncodeRecord serializes (k, data) into a complete record, compressing the payload with zstd when that makes it strictly smaller. k is written as given; canonical-form validation happens on the read side.

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader decodes a wire pack stream.

func NewReader

func NewReader(r io.Reader) *Reader

NewReader returns a Reader over r.

func (*Reader) All

func (r *Reader) All() iter.Seq2[fstree.Object, error]

All iterates over the objects in the stream. It yields exactly one error (and stops) on any structural problem; on a clean stream it yields every object and returns after the end marker. All must be called at most once per Reader because the underlying stream position is not reset between calls.

type Record

type Record struct {
	Key key.Key
	// Flags is the raw record flag byte; pass it to DecodePayload unchanged.
	Flags byte
	Ulen  uint32
	Slen  uint32
}

Record describes a parsed record header. The payload lives at [RecHeaderSize : RecHeaderSize+Slen] within the record's bytes.

func ParseRecord

func ParseRecord(b []byte) (Record, error)

ParseRecord validates the record at the start of b (which may extend past it) and returns its header. It checks framing, flags, key canonicality, and the CRC, without mutating b (b may be a read-only mmap).

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer serializes fstree.Objects into the wire pack format. It is not safe for concurrent use; a client wanting parallel uploads creates one Writer per pack.

func NewWriter

func NewWriter(w io.Writer) *Writer

NewWriter returns a Writer emitting to w. The caller owns w and must close it; Writer.Close only writes the end marker and flushes.

func (*Writer) Add

func (w *Writer) Add(o fstree.Object) error

Add appends one object record.

func (*Writer) AddRecord

func (w *Writer) AddRecord(rec []byte) error

AddRecord appends a pre-encoded record (an EncodeRecord output, as stored verbatim on disk) without decoding or re-encoding it. It is the zero-copy counterpart to Add: the push path reads a record straight from the local store and writes it to the wire, skipping the decompress/recompress round trip. rec is written as given; its framing and CRC are validated by the receiving Reader.

func (*Writer) Close

func (w *Writer) Close() error

Close writes the header (if no object was added) and the end marker, then flushes. It does not close the underlying writer.