volume

Documentation

Overview

Package volume implements the physical storage engine for the blobstore.

Design principles

1. Mechanical sympathy — 88-byte pageHeader, largest fields first, no implicit padding. magicFlags packs sentinel + PageFlags in one word.

2. Minimal syscalls — one Write per page (pooled assembly buffer). ReadChunk uses ReadAt per page. MarkDeleted: ReadAt + WriteAt on the 4-byte magicFlags word. WAL file kept open beside the segment (eliminates os.OpenFile + syscall.ByteSliceFromString per blob).

3. Zero-allocation hot paths: - chunkBuf, pageBuf, sha256 hasher from sync.Pool - blobHasher.Sum into a stack array — no heap digest slice - BlobID encoded from stack buffer, converted directly to string - ChunkID is the chunk's content hash, encoded from a stack buffer - patches slice pre-allocated cap(1); chunks slice pre-allocated cap(1) - WAL write buffer grown once per blob, from pool

4. Lock scope minimised — io.Reader streaming, hashing, and CRC happen with NO lock held. Lock acquired only for bw.Write(pageBuf) per chunk.

PageHeader on-disk layout (88 bytes, no implicit padding)

 0  dataLen      8    uint64
 8  chunkID     32    [32]byte
40  blobID      32    [32]byte
72  chunkSeq    4     uint32
76  totalChunks 4     uint32
80  crc32       4     uint32
84  magicFlags  4     upper 24: 0xB10B5E  lower 8: PageFlags
                      88 total

Segment file layout

[SegmentFileHeader  128 bytes]
[Page 0 … N]  each exactly pageSize bytes: header(88) + payload + padding

WAL entry layout (appended per blob, written to the kept-open wal file)

magic(4) | blobIDLen(2) | blobID | chunkCount(4) |
  per chunk: chunkIDLen(2) | chunkID | segSeq(8) | offset(8) | length(8)

Index

• Constants
• type Engine
  • func Open(rootDir, nsID string, opts Options) (*Engine, error)
  • func (e *Engine) ActiveSegmentID() (id object.SegmentID, ok bool)
  • func (e *Engine) Close() error
  • func (e *Engine) DeleteSegmentFile(segID object.SegmentID) error
  • func (e *Engine) IsDirty() bool
  • func (e *Engine) ListSegmentIDs() ([]object.SegmentID, error)
  • func (e *Engine) MarkDeleted(entry object.ChunkEntry) error
  • func (e *Engine) ParseWAL(segID object.SegmentID) ([]WALEntry, error)
  • func (e *Engine) ReadChunk(entry object.ChunkEntry) ([]byte, error)
  • func (e *Engine) RewriteSegment(oldSegID object.SegmentID) (*SegmentRewriteResult, []object.ChunkEntry, error)
  • func (e *Engine) ScanSegments(fn func(object.ChunkEntry, PageHeader) error) error
  • func (e *Engine) SegmentStats() ([]SegmentStat, error)
  • func (e *Engine) WriteBlob(r io.Reader) (*WriteResult, error)
• type Options
  • func (o Options) Validate() error
• type PageFlags
  • func (f PageFlags) Clear(flag PageFlags) PageFlags
  • func (f PageFlags) IsDeleted() bool
  • func (f PageFlags) IsLastChunk() bool
  • func (f PageFlags) Set(flag PageFlags) PageFlags
• type PageHeader
• type SegmentRewriteResult
• type SegmentStat
  • func (s SegmentStat) DeadRatio() float64
• type WALEntry
• type WriteResult

Constants

const (
	DefaultPageSize       = 16 * 1024
	DefaultChunkSize      = 4 * 1024 * 1024
	DefaultMaxSegmentSize = 512 * 1024 * 1024

	// DefaultSegmentRewriteThreshold is the dead-byte ratio (by page count,
	// 0.0–1.0) a sealed segment must reach before Compact's phase 2
	// physically rewrites it to reclaim space. 0.30 means a segment is
	// rewritten once 30% or more of its pages belong to deleted chunks.
	DefaultSegmentRewriteThreshold = 0.30
)

Default tuning values.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine manages segment files for a single namespace. Safe for concurrent use.

Cache line layout:

• mu + 40-byte explicit pad = exactly 64 bytes (one cache line). Prevents false sharing with active/nextSeq which change per write.

func Open

func Open(rootDir, nsID string, opts Options) (*Engine, error)

Open opens (or creates) a volume engine rooted at rootDir for namespace nsID.

func (*Engine) ActiveSegmentID

func (e *Engine) ActiveSegmentID() (id object.SegmentID, ok bool)

ActiveSegmentID returns the currently-active (still being written to) segment's ID. ok is false if no segment has been created yet. Callers must never rewrite or delete the active segment — RewriteSegment and DeleteSegmentFile both refuse to.

func (*Engine) Close

func (e *Engine) Close() error

Close flushes and closes the active segment, clears the dirty flag.

func (*Engine) DeleteSegmentFile

func (e *Engine) DeleteSegmentFile(segID object.SegmentID) error

DeleteSegmentFile physically removes a sealed segment's data and WAL files. The caller must not call this until every chunk that was live in this segment has had its relocated location durably committed to the index — see RewriteSegment's doc comment and Compact's phase 2 for why that ordering is what keeps a mid-rewrite crash safe. Refuses to delete the currently active segment.

func (*Engine) IsDirty

func (e *Engine) IsDirty() bool

IsDirty reports whether the engine was not cleanly closed on last run.

func (*Engine) ListSegmentIDs

func (e *Engine) ListSegmentIDs() ([]object.SegmentID, error)

ListSegmentIDs returns every segment file's ID by reading the directory listing only — it does not open or read any segment's contents. This is the cheap discovery step WAL replay is built on: finding out which segments exist costs one directory read, versus ScanSegments/ SegmentStats' full page-by-page walk of every byte of every segment file. Replay only needs the small per-segment WAL files after this.

func (*Engine) MarkDeleted

func (e *Engine) MarkDeleted(entry object.ChunkEntry) error

MarkDeleted sets the deleted flag via ReadAt + WriteAt on the magicFlags word.

func (*Engine) ParseWAL

func (e *Engine) ParseWAL(segID object.SegmentID) ([]WALEntry, error)

ParseWAL reads every complete WAL entry recorded for segment segID, in the order they were written, reconstructing a full ChunkEntry for each chunk. PageCount is recomputed via the same ceil-division every other page-count calculation in this package uses (the WAL doesn't store it); Seq is parsed from each ChunkID's own "<blobID>#<seq>" suffix rather than trusted from WAL position, so a caller doesn't have to assume WAL ordering matches chunk ordering — it's cross-checked implicitly by simply not depending on it.

ParseWAL stops cleanly, without error, at the first incomplete or malformed trailing entry, and at a missing WAL file entirely (returns nil, nil). appendWAL writes an entry with a single Write call followed by a Sync; a crash between those two, or mid-Write, can leave a torn partial record as the very last bytes in the file. Every entry before that point was written as a complete unit, so treating a torn tail as "the log simply ends here" is correct — it is indistinguishable from, and handled identically to, "this entry hadn't been synced yet when the process stopped," which is exactly the case a WAL is supposed to leave safely unreplayed.

func (*Engine) ReadChunk

func (e *Engine) ReadChunk(entry object.ChunkEntry) ([]byte, error)

ReadChunk reads and CRC-verifies the payload of a single chunk. RLock — concurrent reads never block each other. One ReadAt per page.

func (*Engine) RewriteSegment

func (e *Engine) RewriteSegment(oldSegID object.SegmentID) (*SegmentRewriteResult, []object.ChunkEntry, error)

RewriteSegment copies every live (not-deleted) chunk in the sealed segment oldSegID, verbatim, into a freshly created segment, and returns the new locations the caller must commit to the index. It does not touch the index itself and does not delete oldSegID's files — package store's Compact owns that ordering (index update, then delete) because getting it backwards is what would make a crash mid-rewrite lose data instead of merely leaving a segment un-reclaimed until the next run.

"Verbatim" matters here: a live chunk's header, payload, and CRC32 are copied as the exact bytes already on disk, not recomputed. The content hasn't changed — only where it lives — so recomputing anything would be pure risk (a transcription bug corrupting an otherwise-untouched chunk) for zero benefit.

Returns an error without creating anything if oldSegID is the currently active segment — it is still being appended to and must never be rewritten out from under a concurrent WriteBlob.

func (*Engine) ScanSegments

func (e *Engine) ScanSegments(fn func(object.ChunkEntry, PageHeader) error) error

ScanSegments iterates every page in every segment file.

func (*Engine) SegmentStats

func (e *Engine) SegmentStats() ([]SegmentStat, error)

SegmentStats scans every segment file and returns per-segment live/dead totals, ordered by SegmentID (equivalently, creation order). It is built on top of ScanSegments — the same page-header deleted flag ReadChunk and MarkDeleted already use — so it stays consistent with them by construction rather than by convention.

This exists to let a caller (package store's Compact) decide which sealed segments are worth physically rewriting, without package volume needing to know anything about the index or about why a chunk became dead — it only reports what the on-disk flags already say.

func (*Engine) WriteBlob

func (e *Engine) WriteBlob(r io.Reader) (*WriteResult, error)

WriteBlob reads all bytes from r, splits into content-defined chunks via FastCDC, and writes them durably. Chunk boundaries are a function of the bytes themselves (see package chunking), and every chunk's identity is its own content hash — so a blob and a blob-plus-a-prefix share their unchanged chunks. Cross-blob reuse of those shared chunks happens at the store layer (index refcounting); this engine always writes every chunk it is given.

Allocation budget per blob:

• 1 × WriteResult escape (unavoidable — caller receives a pointer)
• 1 × []ChunkEntry backing array (result.Chunks)
• 2 × string header per chunk (BlobID + ChunkID — Go string-from-[]byte)
• Pool borrows (pageBuf, hasher, walBuf, chunkIDScratch): zero net

Lock is held only during bw.Write(pageBuf) — not during I/O or hashing.

type Options

type Options struct {
	PageSize       int
	ChunkSize      int64
	MaxSegmentSize int64

	// Cipher, if non-nil, enables encryption-at-rest for every chunk
	// this Engine writes and reads: WriteBlob encrypts each chunk's
	// payload before it reaches disk, and ReadChunk decrypts it after
	// reading and CRC-verifying the stored bytes. Nil (the default)
	// means the namespace is unencrypted — Engine's on-disk behavior is
	// then identical to before this option existed.
	//
	// This is deliberately an Engine-level (i.e. per-namespace) switch,
	// not a global one: package store resolves each namespace's own
	// Cipher independently, from that namespace's own key, so
	// encryption is opt-in per namespace.
	Cipher *encryption.Cipher
}

Options configures a volume Engine.

func (Options) Validate

func (o Options) Validate() error

Validate reports whether o is safe to use. A zero field is never rejected — it means "use the package default", and the defaults are always valid — but an explicitly-set nonsensical value is rejected before it can reach the paging/segment code and cause a panic or silent corruption at runtime. This is deliberately called both here (via Open) and by store.Open, so a bad Config fails immediately, before any disk I/O, regardless of whether the caller goes through package store or uses package volume directly.

type PageFlags

type PageFlags uint8

PageFlags occupies the lower 8 bits of the magicFlags word.

0  FlagDeleted    unreferenced; eligible for compaction
1  FlagLastChunk  final chunk of the blob
2  FlagCompressed reserved

const (
	FlagDeleted    PageFlags = 1 << 0
	FlagLastChunk  PageFlags = 1 << 1
	FlagCompressed PageFlags = 1 << 2
)

func (PageFlags) Clear

func (f PageFlags) Clear(flag PageFlags) PageFlags

func (PageFlags) IsDeleted

func (f PageFlags) IsDeleted() bool

func (PageFlags) IsLastChunk

func (f PageFlags) IsLastChunk() bool

func (PageFlags) Set

func (f PageFlags) Set(flag PageFlags) PageFlags

type PageHeader

type PageHeader struct {
	ChunkID     [32]byte
	BlobID      [32]byte
	ChunkSeq    uint32
	TotalChunks uint32
	DataLen     uint64
	Flags       PageFlags
}

PageHeader is the exported view used in ScanSegments callbacks.

type SegmentRewriteResult

type SegmentRewriteResult struct {
	OldSegmentID object.SegmentID
	NewSegmentID object.SegmentID
	ChunksKept   int   // live chunks copied into the new segment
	PagesFreed   int   // pages belonging to deleted chunks, not copied forward
	BytesFreed   int64 // payload bytes belonging to deleted chunks, not copied forward
}

SegmentRewriteResult summarises the outcome of rewriting one segment.

type SegmentStat

type SegmentStat struct {
	SegmentID  object.SegmentID
	TotalPages int
	DeadPages  int
	TotalBytes int64
	DeadBytes  int64
}

SegmentStat summarises one segment's live/dead page and byte counts, as observed from its on-disk page header flags at the time of the scan.

func (SegmentStat) DeadRatio

func (s SegmentStat) DeadRatio() float64

DeadRatio returns the fraction (0.0–1.0) of this segment's pages that belong to deleted chunks. Returns 0 for a segment with no pages at all.

type WALEntry

type WALEntry struct {
	BlobID object.BlobID
	Chunks []object.ChunkEntry // full ChunkEntry, seq order, ready to PutChunk directly
}

WALEntry is one durably-committed WriteBlob group, as recorded in a segment's WAL file: one blob and every chunk WriteBlob wrote for it, written and fsynced as a single unit (see appendWAL's call site in WriteBlob — the segment itself is fsynced first, then the WAL entry covering all of that blob's chunks is written and fsynced in one call).

type WriteResult

type WriteResult struct {
	BlobID    object.BlobID
	TotalSize int64
	Chunks    []object.ChunkEntry
}

WriteResult is returned by WriteBlob after a blob is durably written.