datastore

package
v0.29.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 17, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package datastore provides chunk storage, indexing, and backup catalog management for pxar archives.

The package implements the Proxmox Backup Server data model: backup data is split into chunks, each chunk is stored as a DataBlob (with optional zstd compression, AES-256-GCM encryption, and CRC32 verification), and chunk references are tracked in dynamic or fixed index files.

Chunk Store

ChunkStore manages chunk storage on the local filesystem. Each chunk is identified by its SHA-256 digest and stored under a .chunks directory: 

store, _ := datastore.NewChunkStore("/backup/datastore")
inserted, size, _ := store.InsertChunk(digest, blobData)
blobData, _ := store.LoadChunk(digest)

Data Blobs

All chunk data is wrapped in a DataBlob envelope containing a magic number and CRC32 checksum: 

blob, _ := datastore.EncodeBlob(rawChunk)
decoded, _ := datastore.DecodeBlob(blob.Bytes())

Use EncodeCompressedBlob for zstd compression, EncodeEncryptedBlob for AES-256-GCM encryption.

Index Files

Dynamic indexes (.didx) map variable-size chunks (from buzhash chunking) to their digests and offsets: 

writer := datastore.NewDynamicIndexWriter(time.Now().Unix())
writer.Add(offset, digest)
indexData, _ := writer.Finish()

reader, _ := datastore.ParseDynamicIndex(indexData)
info, _ := reader.ChunkInfo(0) // info.Start, info.End, info.Digest

Fixed indexes (.fidx) are used for fixed-size chunks (e.g., raw disk images).

Restoration

Restorer reconstructs files from chunk indexes using a ChunkSource: 

restorer := datastore.NewRestorer(chunkSource)
restorer.RestoreFile(idx, writer)
restorer.RestoreRange(idx, offset, length, writer)

Backup Catalogs

BuildCatalogFast performs parallel catalog extraction from a DIDX. BuildDirIndex builds a directory index from goodbye table entries. OnDemandCatalog provides lazy catalog loading from chunked data.

BackupType, BackupGroup, BackupDir, and BackupInfo model the PBS backup namespace hierarchy (type/id/timestamp). Manifest tracks all files in a backup snapshot.

Index

Constants

View Source 
const (
	BlobHeaderSize          = 12 // magic(8) + crc32(4)
	EncryptedBlobHeaderSize = 44 // magic(8) + crc32(4) + iv(16) + tag(16)
	IndexHeaderSize         = 4096
	DynamicEntrySize        = 40 // end_offset(8) + digest(32)
	FixedDigestSize         = 32
	MaxBlobSize             = 128 * 1024 * 1024 // 128MB
)
View Source 
const (
	CatalogEntryDir      byte = 'd'
	CatalogEntryFile     byte = 'f'
	CatalogEntrySymlink  byte = 'l'
	CatalogEntryHardlink byte = 'h'
	CatalogEntryBlockDev byte = 'b'
	CatalogEntryCharDev  byte = 'c'
	CatalogEntryFifo     byte = 'p'
	CatalogEntrySocket   byte = 's'
)

Variables

View Source 
var (
	MagicUncompressedBlob  = [8]byte{66, 171, 56, 7, 190, 131, 112, 161}
	MagicCompressedBlob    = [8]byte{49, 185, 88, 66, 111, 182, 163, 127}
	MagicEncryptedBlob     = [8]byte{123, 103, 133, 190, 34, 45, 76, 240}
	MagicEncrComprBlob     = [8]byte{230, 89, 27, 191, 11, 191, 216, 11}
	MagicFixedChunkIndex   = [8]byte{47, 127, 65, 237, 145, 253, 15, 205}
	MagicDynamicChunkIndex = [8]byte{28, 145, 78, 165, 25, 186, 179, 205}
)

Magic numbers from Proxmox Backup Server (file_formats.rs).

View Source 
var BlobBufPool = sync.Pool{
	New: func() any {
		buf := make([]byte, 0, 4<<20)
		return &buf
	},
}
View Source 
var CatalogMagic = [8]byte{145, 253, 96, 249, 196, 103, 88, 213}
View Source 
var ErrUnknownBlobMagic = fmt.Errorf("unknown blob magic")

ErrUnknownBlobMagic is returned when blob data has an unrecognized magic number.

Functions

func BlobHeaderSizeFor 

func BlobHeaderSizeFor(magic [8]byte) (int, error)

BlobHeaderSizeFor returns the header size for the given blob magic. Returns ErrUnknownBlobMagic for unknown magic values.

func CreateRandomKey added in v0.6.0 

func CreateRandomKey() ([32]byte, error)

CreateRandomKey generates a random 32-byte encryption key.

func DecodeBlob 

func DecodeBlob(raw []byte) ([]byte, error)

DecodeBlob decodes a raw blob, verifies CRC, and returns the payload data. For encrypted blobs, use DecodeEncryptedBlob with a CryptConfig.

func DecodeBlobInto added in v0.3.2 

func DecodeBlobInto(dst []byte, raw []byte) ([]byte, error)

DecodeBlobInto decodes a raw blob into dst, verifying CRC. For compressed blobs, dst is used as the decompression output buffer (grown if needed). For uncompressed blobs, returns a slice into raw (zero allocation). For encrypted blobs, use DecodeEncryptedBlob with a CryptConfig.

func DecodeEncryptedBlob added in v0.6.0 

func DecodeEncryptedBlob(raw []byte, cc *CryptConfig) ([]byte, error)

DecodeEncryptedBlob decodes an encrypted blob and returns the decrypted payload.

func EncodeBlobTo added in v0.3.2 

func EncodeBlobTo(dst []byte, data []byte) ([]byte, error)

EncodeBlobTo encodes data as an uncompressed blob into dst, which must have capacity of at least BlobHeaderSize+len(data). Returns the slice of dst containing the encoded blob. This avoids the DataBlob wrapper allocation.

func EncodeCompressedBlobTo added in v0.3.2 

func EncodeCompressedBlobTo(dst []byte, data []byte) ([]byte, error)

EncodeCompressedBlobTo encodes data as a compressed blob into dst. If compression doesn't reduce size, falls back to uncompressed format. Returns the slice of dst containing the encoded blob.

func EncodeEncryptedBlobTo added in v0.6.0 

func EncodeEncryptedBlobTo(dst []byte, data []byte, cc *CryptConfig, compress bool) ([]byte, error)

EncodeEncryptedBlobTo encodes an encrypted blob into dst.

func FormatFingerprint added in v0.6.0 

func FormatFingerprint(fp [32]byte) string

FormatFingerprint formats a 32-byte fingerprint as colon-separated hex, matching PBS's fingerprint format (uppercase hex with colons).

func GenerateKeyFile added in v0.6.0 

func GenerateKeyFile(password string) ([]byte, error)

GenerateKeyFile creates a new encrypted key file protected by a password. This generates a random 32-byte key and encrypts it with AES-256-GCM using a key derived from the password via PBKDF2.

func IsCompressedMagic 

func IsCompressedMagic(magic [8]byte) bool

IsCompressedMagic returns true for compressed blob types.

func IsEncryptedMagic 

func IsEncryptedMagic(magic [8]byte) bool

IsEncryptedMagic returns true for encrypted blob types.

func PutBlobBuf added in v0.13.0 

func PutBlobBuf(bp *[]byte)

PutBlobBuf resets buf and returns it to the shared pool.

func SignManifest added in v0.6.0 

func SignManifest(manifest *Manifest, cc *CryptConfig) error

SignManifest signs a manifest JSON using HMAC-SHA256 with the id_key. The signature is computed over the canonical JSON with "signature" and "unprotected" fields removed, matching PBS behavior.

func VerifyManifestSignature added in v0.6.0 

func VerifyManifestSignature(manifest *Manifest, cc *CryptConfig) error

VerifyManifestSignature verifies the manifest signature.

Types

type BackupDir 

type BackupDir struct {
	Timestamp time.Time
	Group     BackupGroup
}

BackupDir represents a single backup snapshot.

func (BackupDir) FullPath 

func (d BackupDir) FullPath() string

FullPath returns the absolute path under the base directory.

func (BackupDir) Info 

func (d BackupDir) Info() (*BackupInfo, error)

Info returns detailed information about this backup snapshot.

func (BackupDir) Path 

func (d BackupDir) Path() string

Path returns the relative path (e.g., "vm/100/2023-11-14T22:13:20Z").

type BackupFileInfo added in v0.24.0 

type BackupFileInfo struct {
	Filename  string `json:"filename"`
	CryptMode string `json:"crypt-mode,omitempty"`
	CSum      string `json:"csum"`
	Size      uint64 `json:"size"`
}

BackupFileInfo describes a file in a backup manifest.

type BackupGroup 

type BackupGroup struct {
	ID   string
	Base string
	Type BackupType
}

BackupGroup represents a collection of backup snapshots (e.g., vm/100).

func (BackupGroup) Destroy 

func (g BackupGroup) Destroy() error

Destroy removes the backup group directory.

func (BackupGroup) FullPath 

func (g BackupGroup) FullPath() string

FullPath returns the absolute path under the base directory.

func (BackupGroup) ListSnapshots 

func (g BackupGroup) ListSnapshots(fn func(BackupDir) error) error

ListSnapshots invokes fn for each backup snapshot in this group.

func (BackupGroup) Path 

func (g BackupGroup) Path() string

Path returns the relative path for this group (e.g., "vm/100").

type BackupInfo 

type BackupInfo struct {
	Dir       BackupDir
	Files     []string
	Protected bool
}

BackupInfo holds metadata about a backup snapshot.

type BackupType 

type BackupType int

BackupType identifies the kind of backup. 

const (
	BackupVM BackupType = iota
	BackupCT
	BackupHost
)

func ParseBackupType 

func ParseBackupType(s string) (BackupType, error)

ParseBackupType parses a backup type string.

func (BackupType) String 

func (bt BackupType) String() string

type BlobHeader 

type BlobHeader struct {
	Magic [8]byte
	CRC   uint32
}

BlobHeader is the 12-byte header for uncompressed and compressed blobs.

func UnmarshalBlobHeader 

func UnmarshalBlobHeader(data []byte) (BlobHeader, error)

UnmarshalBlobHeader parses a BlobHeader from raw bytes.

func (*BlobHeader) MarshalTo 

func (h *BlobHeader) MarshalTo(buf []byte)

MarshalTo writes the header to buf (must be at least BlobHeaderSize bytes).

type BuildResult added in v0.12.1 

type BuildResult struct {
	Index        *DirIndex
	RootChildren []CatalogChild
}

BuildResult is returned by BuildDirIndex. It contains the lightweight DirIndex and, as a free side effect of the scan, the root directory's direct children.

func BuildDirIndex added in v0.12.0 

func BuildDirIndex(
	metaIdx *DynamicIndexReader,
	source ChunkSource,
	opts CatalogOptions,
) (*BuildResult, error)

BuildDirIndex builds a lightweight directory location index by scanning metadata chunks on demand. It fetches and processes one chunk at a time with a background prefetch goroutine, keeping at most 2-3 decoded chunks in memory regardless of total metadata size.

Peak memory: O(directory_count × avg_path_length) for the index + 2–3 chunks (~8–12 MB for 4 MB chunks) during the scan.

type Catalog added in v0.11.0 

type Catalog struct {
	Dirs map[string][]CatalogChild
	// contains filtered or unexported fields
}

Catalog is a lightweight directory tree: parentPath → children.

The chunks field retains decoded chunk data so that filename strings (created via unsafe.String during parsing) remain valid for the Catalog's lifetime. Callers should not retain a Catalog longer than needed.

func BuildCatalogFast added in v0.11.0 

func BuildCatalogFast(
	metaIdx *DynamicIndexReader,
	source ChunkSource,
	opts CatalogOptions,
) (*Catalog, error)

BuildCatalogFast downloads metadata chunks in parallel and performs a single sequential pass to build a directory catalog — no concatenation. Chunks are parsed in-place; filename strings are zero-copy views into chunk data kept alive by Catalog.chunks.

type CatalogChild added in v0.11.0 

type CatalogChild struct {
	Name string
	Size int64
	Kind pxar.EntryKind
}

CatalogChild is a lightweight directory entry. Field order: largest first to minimize padding (Size int64, Name string, Kind pxar.EntryKind). Total: 8 + 16 + 8 = 32 bytes.

type CatalogEntryType added in v0.6.0 

type CatalogEntryType int
const (
	CatalogEntryTypeFile CatalogEntryType = iota
	CatalogEntryTypeDir
	CatalogEntryTypeSymlink
	CatalogEntryTypeHardlink
	CatalogEntryTypeBlockDev
	CatalogEntryTypeCharDev
	CatalogEntryTypeFifo
	CatalogEntryTypeSocket
)

type CatalogOptions added in v0.11.0 

type CatalogOptions struct {
	MaxWorkers int // parallel chunk downloads (default 4)
}

CatalogOptions configures BuildCatalogFast behavior.

type CatalogReader added in v0.6.0 

type CatalogReader struct {
	// contains filtered or unexported fields
}

func NewCatalogReader added in v0.6.0 

func NewCatalogReader(data []byte) *CatalogReader

type CatalogTreeEntry added in v0.6.0 

type CatalogTreeEntry struct {
	Name      string
	Children  []CatalogTreeEntry
	EntryType CatalogEntryType
	Size      uint64
	Mtime     int64
}

func ReadCatalogTree added in v0.6.0 

func ReadCatalogTree(data []byte) (*CatalogTreeEntry, error)

type CatalogWriter added in v0.6.0 

type CatalogWriter struct {
	// contains filtered or unexported fields
}

CatalogWriter produces a pcat1 binary catalog stream compatible with Proxmox Backup Server. Directories are written bottom-up: leaf directories before their parents. Each directory block contains entries followed by a trailer with metadata. The stream ends with an 8-byte little-endian root directory offset.

Block format: [entries...][entry_count(u64)][name_len(u64)][name(bytes)] Entry format: [name_len(u64)][type(1)][name(bytes)][type-specific data...] File extra: [size(u64)][mtime(i64)] Dir extra: [offset(u64)] (offset from dir start in the stream)

func NewCatalogWriter added in v0.6.0 

func NewCatalogWriter(w io.Writer) *CatalogWriter

NewCatalogWriter creates a catalog writer and writes the pcat1 magic header.

func (*CatalogWriter) AddBlockDevice added in v0.6.0 

func (cw *CatalogWriter) AddBlockDevice(name string)

func (*CatalogWriter) AddCharDevice added in v0.6.0 

func (cw *CatalogWriter) AddCharDevice(name string)

func (*CatalogWriter) AddFIFO added in v0.18.0 

func (cw *CatalogWriter) AddFIFO(name string)

func (*CatalogWriter) AddFile added in v0.6.0 

func (cw *CatalogWriter) AddFile(name string, size uint64, mtime int64)
func (cw *CatalogWriter) AddHardlink(name string)

func (*CatalogWriter) AddSocket added in v0.6.0 

func (cw *CatalogWriter) AddSocket(name string)
func (cw *CatalogWriter) AddSymlink(name string)

func (*CatalogWriter) EndDirectory added in v0.6.0 

func (cw *CatalogWriter) EndDirectory()

EndDirectory encodes the current directory block and adds a Dir entry to the parent directory.

func (*CatalogWriter) Finish added in v0.6.0 

func (cw *CatalogWriter) Finish() error

Finish encodes the root directory block and writes the 8-byte little-endian root directory offset at the end of the stream.

func (*CatalogWriter) StartDirectory added in v0.6.0 

func (cw *CatalogWriter) StartDirectory(name string)

type ChunkInfo 

type ChunkInfo struct {
	Start  uint64
	End    uint64
	Digest [32]byte
}

ChunkInfo describes a single chunk's position and digest.

type ChunkResult 

type ChunkResult struct {
	Digest [32]byte // SHA-256 of raw chunk data
	Offset uint64   // start offset in the original stream
	Size   int      // chunk data size in bytes
	Exists bool     // true if chunk was already in the store
}

ChunkResult describes a single chunk produced by the chunker pipeline.

type ChunkSource 

type ChunkSource interface {
	// GetChunk retrieves a chunk by its SHA-256 digest.
	// Returns the raw chunk data (not decoded/blob-wrapped).
	GetChunk(digest [32]byte) ([]byte, error)
}

ChunkSource provides access to chunks by their digest.

type ChunkStore 

type ChunkStore struct {
	// contains filtered or unexported fields
}

ChunkStore manages chunk storage on the filesystem. Chunks are stored under base/.chunks/XXXX/XXXXYY... where XXXX are the first four hex characters of the SHA-256 digest (PBS-compatible layout).

func NewChunkStore 

func NewChunkStore(base string) (*ChunkStore, error)

NewChunkStore creates a ChunkStore rooted at base, creating the .chunks directory if needed.

func (*ChunkStore) ChunkPath 

func (cs *ChunkStore) ChunkPath(digest [32]byte) string

ChunkPath returns the filesystem path for a chunk identified by digest.

func (*ChunkStore) InsertChunk 

func (cs *ChunkStore) InsertChunk(digest [32]byte, data []byte) (bool, int, error)

InsertChunk stores a chunk. Returns (exists, size, error). If the chunk already exists, returns (true, existingSize, nil).

func (*ChunkStore) LoadChunk 

func (cs *ChunkStore) LoadChunk(digest [32]byte) ([]byte, error)

LoadChunk reads a chunk from disk.

func (*ChunkStore) TouchChunk 

func (cs *ChunkStore) TouchChunk(digest [32]byte) error

TouchChunk updates the access time of a chunk file.

type ChunkStoreSource 

type ChunkStoreSource struct {
	// contains filtered or unexported fields
}

ChunkStoreSource adapts a ChunkStore to the ChunkSource interface.

func NewChunkStoreSource 

func NewChunkStoreSource(store *ChunkStore) *ChunkStoreSource

NewChunkStoreSource creates a chunk source from a local chunk store.

func (*ChunkStoreSource) GetChunk 

func (s *ChunkStoreSource) GetChunk(digest [32]byte) ([]byte, error)

GetChunk retrieves a chunk from the local store.

type CryptConfig added in v0.6.0 

type CryptConfig struct {
	// contains filtered or unexported fields
}

CryptConfig holds the derived keys needed for encryption, signing, and fingerprinting.

func LoadKeyFile added in v0.6.0 

func LoadKeyFile(data []byte, password string) (*CryptConfig, error)

LoadKeyFile decrypts a key file using a password and returns the raw encryption key.

func LoadKeyFileNoPassword added in v0.6.0 

func LoadKeyFileNoPassword(data []byte) (*CryptConfig, error)

LoadKeyFileNoPassword loads a key file with "none" KDF (no encryption).

func NewCryptConfig added in v0.6.0 

func NewCryptConfig(encKey [32]byte) (*CryptConfig, error)

NewCryptConfig derives signing and fingerprint keys from a raw 32-byte encryption key.

func (*CryptConfig) AuthTag added in v0.6.0 

func (c *CryptConfig) AuthTag(data []byte) [32]byte

AuthTag computes an HMAC-SHA256 authentication tag over data using the id_key.

func (*CryptConfig) ComputeDigest added in v0.6.0 

func (c *CryptConfig) ComputeDigest(data []byte) [32]byte

ComputeDigest computes SHA-256(data || id_key), matching PBS compute_digest.

func (*CryptConfig) Decrypt added in v0.6.0 

func (c *CryptConfig) Decrypt(data []byte) ([]byte, error)

Decrypt decrypts data encrypted by Encrypt. Input format: nonce || ciphertext (with GCM tag appended).

func (*CryptConfig) Encrypt added in v0.6.0 

func (c *CryptConfig) Encrypt(plaintext []byte) ([]byte, error)

Encrypt encrypts plaintext using AES-256-GCM with a random nonce. Returns nonce + ciphertext (ciphertext includes the GCM tag).

func (*CryptConfig) Fingerprint added in v0.6.0 

func (c *CryptConfig) Fingerprint() [32]byte

Fingerprint computes SHA-256(fingerprintInput || id_key), matching PBS key fingerprint.

type CryptMode added in v0.6.0 

type CryptMode string

CryptMode represents the encryption mode for a backup. 

const (
	CryptModeNone    CryptMode = "none"      // No encryption (default)
	CryptModeEncrypt CryptMode = "encrypt"   // AEAD encryption
	CryptModeSign    CryptMode = "sign-only" // Sign only (no encryption)
)

type DataBlob 

type DataBlob struct {
	// contains filtered or unexported fields
}

DataBlob represents a stored data blob with optional compression. The raw data contains the magic, CRC, and payload.

func EncodeBlob 

func EncodeBlob(data []byte) (*DataBlob, error)

EncodeBlob creates an uncompressed blob from data.

func EncodeCompressedBlob 

func EncodeCompressedBlob(data []byte) (*DataBlob, error)

EncodeCompressedBlob creates a compressed blob. Falls back to uncompressed if compression doesn't reduce size.

func EncodeEncryptedBlob added in v0.6.0 

func EncodeEncryptedBlob(data []byte, cc *CryptConfig, compress bool) (*DataBlob, error)

EncodeEncryptedBlob creates an encrypted blob from data using AES-256-GCM. If compress is true and the data compresses well, it is compressed before encryption (producing an EncrCompr blob). Otherwise, an Encrypted blob is produced.

func (*DataBlob) Bytes 

func (b *DataBlob) Bytes() []byte

Bytes returns the raw blob bytes (header + payload).

func (*DataBlob) IsCompressed 

func (b *DataBlob) IsCompressed() bool

IsCompressed returns true if the blob uses compression.

func (*DataBlob) IsEncrypted 

func (b *DataBlob) IsEncrypted() bool

IsEncrypted returns true if the blob uses encryption.

func (*DataBlob) Magic 

func (b *DataBlob) Magic() [8]byte

Magic returns the blob magic number.

type DirIndex added in v0.12.0 

type DirIndex struct {
	// contains filtered or unexported fields
}

DirIndex is a lightweight mapping from directory paths to their positions in the metadata stream. Use with OnDemandCatalog for lazy directory loading.

Memory: ~80 bytes per directory (path string + location struct). For 1M directories: ~80 MB (vs 2+ GB for a full Catalog).

func (*DirIndex) DirPaths added in v0.12.0 

func (idx *DirIndex) DirPaths(fn func(string) error) error

DirPaths invokes fn for each known directory path in unspecified order.

func (*DirIndex) HasDir added in v0.12.0 

func (idx *DirIndex) HasDir(path string) bool

HasDir reports whether a directory path exists in the index.

func (*DirIndex) NumDirs added in v0.12.0 

func (idx *DirIndex) NumDirs() int

NumDirs returns the number of directories in the index.

type DynamicEntry 

type DynamicEntry struct {
	EndOffset uint64
	Digest    [32]byte
}

DynamicEntry is a single entry in a dynamic index (40 bytes).

type DynamicIndexHeader 

type DynamicIndexHeader struct {
	Magic     [8]byte
	UUID      [16]byte
	Ctime     int64
	IndexCsum [32]byte
}

DynamicIndexHeader is the 4096-byte header for dynamic chunk index files.

func UnmarshalDynamicIndexHeader 

func UnmarshalDynamicIndexHeader(data []byte) (DynamicIndexHeader, error)

UnmarshalDynamicIndexHeader parses a DynamicIndexHeader from raw bytes.

func (*DynamicIndexHeader) MarshalTo 

func (h *DynamicIndexHeader) MarshalTo(buf []byte)

MarshalTo writes the header to buf (must be at least IndexHeaderSize bytes).

type DynamicIndexReader 

type DynamicIndexReader struct {
	// contains filtered or unexported fields
}

DynamicIndexReader reads a dynamic chunk index.

func ParseDynamicIndex added in v0.18.0 

func ParseDynamicIndex(data []byte) (*DynamicIndexReader, error)

ParseDynamicIndex parses a dynamic index from raw bytes.

func (*DynamicIndexReader) CTime 

func (r *DynamicIndexReader) CTime() int64

CTime returns the creation timestamp.

func (*DynamicIndexReader) ChunkFromOffset 

func (r *DynamicIndexReader) ChunkFromOffset(offset uint64) (int, bool)

ChunkFromOffset returns the chunk index containing the given byte offset. Uses binary search for O(log n) lookup.

func (*DynamicIndexReader) ChunkInfo 

func (r *DynamicIndexReader) ChunkInfo(pos int) (ChunkInfo, bool)

ChunkInfo returns the chunk info at position i.

func (*DynamicIndexReader) ComputeCsum 

func (r *DynamicIndexReader) ComputeCsum() ([32]byte, uint64)

ComputeCsum computes the SHA-256 checksum over all entry data.

func (*DynamicIndexReader) Count 

func (r *DynamicIndexReader) Count() int

Count returns the number of entries.

func (*DynamicIndexReader) Entry 

func (r *DynamicIndexReader) Entry(i int) DynamicEntry

Entry returns the entry at position i.

func (*DynamicIndexReader) IndexBytes 

func (r *DynamicIndexReader) IndexBytes() uint64

IndexBytes returns the total virtual size (end offset of last entry).

func (*DynamicIndexReader) IndexDigest 

func (r *DynamicIndexReader) IndexDigest(pos int) ([32]byte, bool)

IndexDigest returns the digest at position pos.

type DynamicIndexWriter 

type DynamicIndexWriter struct {
	// contains filtered or unexported fields
}

DynamicIndexWriter builds a dynamic chunk index.

func NewDynamicIndexWriter 

func NewDynamicIndexWriter(ctime int64) *DynamicIndexWriter

NewDynamicIndexWriter creates a new writer with the given creation time.

func (*DynamicIndexWriter) Add 

func (w *DynamicIndexWriter) Add(endOffset uint64, digest [32]byte)

Add appends an entry with the given end offset and digest.

func (*DynamicIndexWriter) Csum added in v0.3.0 

func (w *DynamicIndexWriter) Csum() [32]byte

Csum returns the SHA-256 checksum over all entry data (end_offset || digest pairs). This matches PBS's compute_csum() and is the checksum stored in the manifest. The result is cached and invalidated by Add().

func (*DynamicIndexWriter) Finish 

func (w *DynamicIndexWriter) Finish() ([]byte, error)

Finish writes the complete index and returns the raw bytes.

func (*DynamicIndexWriter) LastEndOffset added in v0.27.2 

func (w *DynamicIndexWriter) LastEndOffset() uint64

LastEndOffset returns the end offset of the last entry, which is the total virtual size of the indexed stream. Returns 0 if there are no entries.

type EncryptedBlobHeader 

type EncryptedBlobHeader struct {
	Magic [8]byte
	CRC   uint32
	IV    [16]byte
	Tag   [16]byte
}

EncryptedBlobHeader is the 48-byte header for encrypted blobs.

func UnmarshalEncryptedBlobHeader 

func UnmarshalEncryptedBlobHeader(data []byte) (EncryptedBlobHeader, error)

UnmarshalEncryptedBlobHeader parses an EncryptedBlobHeader from raw bytes.

func (*EncryptedBlobHeader) MarshalTo 

func (h *EncryptedBlobHeader) MarshalTo(buf []byte)

MarshalTo writes the header to buf (must be at least EncryptedBlobHeaderSize bytes).

type FixedIndexHeader 

type FixedIndexHeader struct {
	Magic     [8]byte
	UUID      [16]byte
	Ctime     int64
	IndexCsum [32]byte
	Size      uint64
	ChunkSize uint64
}

FixedIndexHeader is the 4096-byte header for fixed chunk index files.

func UnmarshalFixedIndexHeader 

func UnmarshalFixedIndexHeader(data []byte) (FixedIndexHeader, error)

UnmarshalFixedIndexHeader parses a FixedIndexHeader from raw bytes.

func (*FixedIndexHeader) MarshalTo 

func (h *FixedIndexHeader) MarshalTo(buf []byte)

MarshalTo writes the header to buf (must be at least IndexHeaderSize bytes).

type FixedIndexReader 

type FixedIndexReader struct {
	// contains filtered or unexported fields
}

FixedIndexReader reads a fixed-size chunk index.

func ParseFixedIndex added in v0.18.0 

func ParseFixedIndex(data []byte) (*FixedIndexReader, error)

ParseFixedIndex parses a fixed index from raw bytes.

func (*FixedIndexReader) CTime 

func (r *FixedIndexReader) CTime() int64

CTime returns the creation timestamp.

func (*FixedIndexReader) ChunkFromOffset 

func (r *FixedIndexReader) ChunkFromOffset(offset uint64) (int, bool)

ChunkFromOffset returns the chunk index for the given byte offset.

func (*FixedIndexReader) ChunkInfo 

func (r *FixedIndexReader) ChunkInfo(pos int) (ChunkInfo, bool)

ChunkInfo returns chunk info at position pos.

func (*FixedIndexReader) ComputeCsum 

func (r *FixedIndexReader) ComputeCsum() ([32]byte, uint64)

ComputeCsum computes the SHA-256 checksum over all digests.

func (*FixedIndexReader) Count 

func (r *FixedIndexReader) Count() int

Count returns the number of chunks.

func (*FixedIndexReader) IndexBytes 

func (r *FixedIndexReader) IndexBytes() uint64

IndexBytes returns the total virtual size.

func (*FixedIndexReader) IndexDigest 

func (r *FixedIndexReader) IndexDigest(pos int) ([32]byte, bool)

IndexDigest returns the digest at position pos.

type FixedIndexWriter 

type FixedIndexWriter struct {
	// contains filtered or unexported fields
}

FixedIndexWriter builds a fixed-size chunk index.

func NewFixedIndexWriter 

func NewFixedIndexWriter(ctime int64, size, chunkSize uint64) (*FixedIndexWriter, error)

NewFixedIndexWriter creates a writer. ChunkSize must be a power of 2.

func (*FixedIndexWriter) Finish 

func (w *FixedIndexWriter) Finish() ([]byte, error)

Finish writes the complete index and returns raw bytes.

func (*FixedIndexWriter) Set 

func (w *FixedIndexWriter) Set(i int, digest [32]byte)

Set sets the digest for chunk at index i.

type KeyConfig added in v0.6.0 

type KeyConfig struct {
	Created     string              `json:"created,omitempty"`
	Modified    string              `json:"modified,omitempty"`
	Fingerprint string              `json:"fingerprint,omitempty"`
	Data        []byte              `json:"data"`
	Kdf         KeyDerivationConfig `json:"kdf"`
}

KeyConfig represents an encryption key file that can be stored on disk.

type KeyDerivationConfig added in v0.6.0 

type KeyDerivationConfig struct {
	Type string `json:"type"` // "scrypt" or "pbkdf2" or "none"
	Salt []byte `json:"salt,omitempty"`
	N    int    `json:"n,omitempty"`    // scrypt: CPU cost
	R    int    `json:"r,omitempty"`    // scrypt: block size
	P    int    `json:"p,omitempty"`    // scrypt: parallelism
	Iter int    `json:"iter,omitempty"` // pbkdf2: iterations
}

KeyDerivationConfig specifies the key derivation function parameters.

type Manifest 

type Manifest struct {
	BackupType  string           `json:"backup-type"`
	BackupID    string           `json:"backup-id"`
	CryptMode   string           `json:"crypt-mode,omitempty"`
	Signature   string           `json:"signature,omitempty"`
	Files       []BackupFileInfo `json:"files"`
	Unprotected json.RawMessage  `json:"unprotected,omitempty"`
	BackupTime  int64            `json:"backup-time"`
}

Manifest represents a backup manifest (index.json).

func UnmarshalManifest 

func UnmarshalManifest(data []byte) (*Manifest, error)

UnmarshalManifest parses a manifest from JSON.

func (*Manifest) Marshal 

func (m *Manifest) Marshal() ([]byte, error)

Marshal serializes the manifest to JSON.

func (*Manifest) VerifyFile 

func (m *Manifest) VerifyFile(filename, csum string, size uint64) error

VerifyFile checks that a file's checksum and size match the manifest.

type OnDemandCatalog added in v0.12.0 

type OnDemandCatalog struct {
	// contains filtered or unexported fields
}

OnDemandCatalog provides lazy directory listing over a metadata stream. Only the lightweight DirIndex (~80 MB for 1M dirs) is held in memory; chunks are fetched from the source when ListDir is called. An LRU cache of decoded chunks avoids re-fetching recently used chunks.

Safe for concurrent use. Internal LRU cache is mutex-protected.

func NewOnDemandCatalog added in v0.12.0 

func NewOnDemandCatalog(index *DirIndex, metaIdx *DynamicIndexReader, source ChunkSource) *OnDemandCatalog

NewOnDemandCatalog creates a lazy catalog backed by the given index and source.

func (*OnDemandCatalog) DirPaths added in v0.12.0 

func (c *OnDemandCatalog) DirPaths(fn func(string) error) error

DirPaths invokes fn for each known directory path.

func (*OnDemandCatalog) HasDir added in v0.12.0 

func (c *OnDemandCatalog) HasDir(path string) bool

HasDir reports whether a directory exists in the index.

func (*OnDemandCatalog) ListDir added in v0.12.0 

func (c *OnDemandCatalog) ListDir(path string, fn func(CatalogChild) error) error

ListDir fetches and parses a single directory's children on demand. Only the chunk(s) containing that directory's entries are fetched. Returned children own their data — safe to retain across calls.

If the directory has nested subdirectories, their subtrees are skipped using end offsets from the DirIndex (when available) or depth tracking (fallback). Only the direct children of the requested directory are returned. For each child, fn is called with a CatalogChild; if fn returns a non-nil error, iteration stops and the error is returned.

func (*OnDemandCatalog) NumDirs added in v0.12.0 

func (c *OnDemandCatalog) NumDirs() int

NumDirs returns the total number of directories.

type Restorer 

type Restorer struct {
	// contains filtered or unexported fields
}

Restorer reconstructs files from dynamic indexes using a chunk source.

func NewRestorer 

func NewRestorer(source ChunkSource) *Restorer

NewRestorer creates a new restorer with the given chunk source.

func (*Restorer) FileSize 

func (r *Restorer) FileSize(idx *DynamicIndexReader) uint64

FileSize returns the total size of the file represented by the index.

func (*Restorer) RestoreFile 

func (r *Restorer) RestoreFile(idx *DynamicIndexReader, w io.Writer) error

RestoreFile reconstructs a complete file from a dynamic index. Writes the reconstructed file content to w.

func (*Restorer) RestoreRange 

func (r *Restorer) RestoreRange(idx *DynamicIndexReader, offset, length uint64, w io.Writer) error

RestoreRange reconstructs a specific byte range from a dynamic index. Useful for partial reads without downloading the entire file.

type StoreChunker 

type StoreChunker struct {
	// contains filtered or unexported fields
}

StoreChunker splits a data stream into variable-size chunks using buzhash content-defined chunking, computes digests, stores chunks via ChunkStore, and builds a DynamicIndexWriter.

func NewStoreChunker 

func NewStoreChunker(store *ChunkStore, config buzhash.Config, compress bool) *StoreChunker

NewStoreChunker creates a chunker pipeline. If compress is true, chunks are stored as compressed DataBlobs; otherwise as uncompressed blobs.

func (*StoreChunker) ChunkStream 

func (sc *StoreChunker) ChunkStream(r io.Reader, fn func(ChunkResult) error) ([]ChunkResult, *DynamicIndexWriter, error)

ChunkStream reads all data from r, stores each chunk, and builds an index. fn is called for each chunk after it is stored; if fn returns a non-nil error, chunking stops and the error is returned. If fn is nil, no callback is made.

type UnprotectedInfo added in v0.6.0 

type UnprotectedInfo struct {
	KeyFingerprint string `json:"key-fingerprint"`
}

UnprotectedInfo holds the unprotected key info in a manifest.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.