transfer

package
v0.17.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 12, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package transfer provides utilities for transferring files between pxar archives.

Transfer implements archive-to-archive copy, directory tree walking, catalog extraction, and streaming read/write adapters for PBS remote stores, local chunk stores, and raw io.Reader/io.Writer streams.

Readers

FileArchiveReader wraps accessor.Accessor for local .pxar files. ChunkedArchiveReader supports lazy on-demand chunk loading from .didx indexes. SplitArchiveReader handles v2 split archives (.mpxar + .ppxar). PBSArchiveReader connects to Proxmox Backup Server via the reader protocol.

Writers

StreamArchiveWriter encodes to io.Writer streams (v1 and v2). ChunkedArchiveWriter chunks and stores via datastore.ChunkStore. SessionArchiveWriter uploads through a backupproxy.BackupSession.

Walks

WalkTree visits every entry in a directory tree with optional content reading. WalkTreeMeta performs metadata-only traversal. WalkFilter selects which entry types to visit.

Package transfer provides utilities for transferring files between pxar archives.

Index

Constants

This section is empty.

Variables

View Source 
var ErrSkipDir = fmt.Errorf("skip directory")

ErrSkipDir can be returned by a WalkFunc to skip a directory's children.

View Source 
var WalkMetaOnly = WalkOption{MetaOnly: true}

WalkMetaOnly is a convenience WalkOption for metadata-only walks.

Functions

func ComputeContentDigest 

func ComputeContentDigest(source datastore.ChunkSource, payloadIdx *datastore.DynamicIndexReader, payloadOffset, fileSize uint64) ([32]byte, error)

ComputeContentDigest computes SHA-256 of a file's content from the source archive without reconstructing the entire payload stream. Only loads the chunks needed for that specific file.

func Copy 

func Copy(src ArchiveReader, dst ArchiveWriter, mappings []PathMapping, opts TransferOption) error

Copy copies files from the source archive to the target writer. Each PathMapping specifies a source path (inside the source archive) and a destination path (inside the target archive). If the source entry is a directory, the entire subtree is copied with paths remapped from Src to Dst prefix. If the source entry is a file, the file is written with its path remapped.

func CopyTree 

func CopyTree(src ArchiveReader, dst ArchiveWriter, srcPath, dstPath string, opts TransferOption) error

CopyTree copies a directory tree from srcPath in the source archive to dstPath in the target. All entries under the source directory have their paths remapped from the srcPath prefix to the dstPath prefix.

func ReadFileContentFromChunks 

func ReadFileContentFromChunks(source datastore.ChunkSource, payloadIdx *datastore.DynamicIndexReader, payloadOffset, fileSize uint64) ([]byte, error)

ReadFileContentFromChunks reads a file's content by loading only the necessary payload chunks. This is more efficient than reconstructing the entire payload stream when you only need specific files.

func WalkTree 

func WalkTree(reader ArchiveReader, rootPath string, fn WalkFunc) error

WalkTree walks a directory tree from an ArchiveReader, calling fn for each entry in encoder-compatible order. For directories, fn is called before children are walked. Content is populated for regular files; for other entry types, content is nil.

If fn returns ErrSkipDir for a directory entry, the directory's children are skipped. If fn returns any other error, walking stops.

func WalkTreeMeta added in v0.9.0 

func WalkTreeMeta(reader ArchiveReader, rootPath string, filter WalkFilter, fn MetaWalkFunc) error

WalkTreeMeta walks a directory tree in metadata-only mode with a simplified callback. Content is never read, and the filter mask controls which entry types are visited. Use WalkAll to visit all types.

func WalkTreeWith added in v0.9.0 

func WalkTreeWith(reader ArchiveReader, rootPath string, opts WalkOption, fn WalkFunc) error

WalkTreeWith walks a directory tree with the given options. When opts.MetaOnly is true, file content is never read and content is always nil. When opts.Filter is non-zero, entries not matching the filter are skipped.

Types

type ArchiveReader 

type ArchiveReader interface {
	// ReadRoot returns the root directory entry.
	ReadRoot() (*pxar.Entry, error)

	// Lookup finds an entry by archive-internal path.
	Lookup(path string) (*pxar.Entry, error)

	// ListDirectory streams directory entries without materializing a slice.
	// For each entry, fn is called with a pointer valid only during the callback.
	// If fn returns a non-nil error, iteration stops.
	ListDirectory(dirOffset int64, opts accessor.ListOption, fn func(*pxar.Entry) error) error

	// ReadFileContent reads the complete content of a regular file.
	ReadFileContent(entry *pxar.Entry) ([]byte, error)

	// ReadFileContentReader returns a streaming reader for file content.
	// The caller must close the reader. Use this for large files to avoid
	// buffering the entire content in memory.
	ReadFileContentReader(entry *pxar.Entry) (io.ReadCloser, error)

	// ReadCatalog streams the full directory tree via a callback with
	// minimal decoding. For each entry, fn is called. If fn returns a
	// non-nil error, iteration stops and the error is returned.
	// Significantly faster than WalkTree for indexing.
	ReadCatalog(fn func(CatalogEntry) error) error

	// Close releases resources.
	Close() error
}

ArchiveReader provides unified read access to any pxar archive format.

type ArchiveWriter 

type ArchiveWriter interface {
	// Begin starts writing to a new archive with the given root metadata.
	Begin(rootMeta *pxar.Metadata, opts WriterOptions) error

	// WriteEntry writes an entry (file, symlink, device, etc.) to the archive.
	// For regular files, content is the file data. For other types, content may be nil.
	WriteEntry(entry *pxar.Entry, content []byte) error

	// WriteEntryRef writes an entry that references existing payload data
	// without writing the payload itself. The payloadOffset is the byte offset
	// in the original payload stream. Used for chunk-level deduplication.
	WriteEntryRef(entry *pxar.Entry, payloadOffset uint64) error

	// WriteEntryReader writes a file entry with content streamed from r.
	// size is the total byte count. For non-file entries (symlink, device,
	// fifo, socket), r and size are ignored and content is nil.
	// The caller must ensure r provides exactly size bytes.
	WriteEntryReader(entry *pxar.Entry, r io.Reader, size uint64) error

	// BeginDirectory pushes a directory context.
	BeginDirectory(name string, meta *pxar.Metadata) error

	// EndDirectory pops a directory context.
	EndDirectory() error

	// Finish finalizes the archive.
	Finish() error

	// Close releases resources.
	Close() error
}

ArchiveWriter provides unified write access to any pxar archive format.

type CatalogEntry added in v0.10.0 

type CatalogEntry struct {
	Path       string
	ParentPath string
	Kind       pxar.EntryKind
	FileSize   uint64
}

CatalogEntry is a stripped-down entry for index-building. It contains only the fields needed for cataloging: path, kind, size, and parent.

type ChunkRange 

type ChunkRange struct {
	ChunkIndex   int
	Digest       [32]byte
	ChunkStart   uint64 // start offset in the payload stream
	ChunkEnd     uint64 // end offset in the payload stream
	ContentStart uint64 // start of overlap with file content
	ContentEnd   uint64 // end of overlap with file content
	IsFullChunk  bool   // true if the entire chunk is within the file's content
}

ChunkRange describes a chunk's overlap with a file's content.

func MapFileToPayloadChunks 

func MapFileToPayloadChunks(payloadIdx *datastore.DynamicIndexReader, payloadOffset, fileSize uint64) []ChunkRange

MapFileToPayloadChunks maps a file's content in the source payload stream to the chunk digests that contain it. This is used to know which source chunks are needed for a specific file without downloading the entire payload stream.

Returns a list of (chunkIndex, digest) pairs for chunks that overlap the file's content range, and the byte offsets within those chunks that correspond to the file content.

type ChunkedArchiveReader 

type ChunkedArchiveReader struct {
	// contains filtered or unexported fields
}

ChunkedArchiveReader reads from a chunked archive (.pxar.didx). It lazily loads chunks on demand using a ChunkedReadSeeker, avoiding full-stream-in-memory reconstruction. For small archives where full reconstruction is acceptable, use NewChunkedArchiveReaderEager.

func NewChunkedArchiveReader 

func NewChunkedArchiveReader(idxData []byte, source datastore.ChunkSource) (*ChunkedArchiveReader, error)

NewChunkedArchiveReader creates a reader for a chunked .pxar.didx archive using lazy on-demand chunk loading. This avoids reconstructing the entire stream into memory — only chunks needed for Lookups and ReadFileContent calls are loaded.

func NewChunkedArchiveReaderEager 

func NewChunkedArchiveReaderEager(idxData []byte, source datastore.ChunkSource) (*ChunkedArchiveReader, error)

NewChunkedArchiveReaderEager creates a reader that reconstructs the entire stream into memory upfront. Use this for small archives or when you need guaranteed sequential access performance.

func (*ChunkedArchiveReader) Close 

func (r *ChunkedArchiveReader) Close() error

func (*ChunkedArchiveReader) ListDirectory 

func (r *ChunkedArchiveReader) ListDirectory(dirOffset int64, opts accessor.ListOption, fn func(*pxar.Entry) error) error

func (*ChunkedArchiveReader) Lookup 

func (r *ChunkedArchiveReader) Lookup(path string) (*pxar.Entry, error)

func (*ChunkedArchiveReader) ReadCatalog added in v0.10.0 

func (r *ChunkedArchiveReader) ReadCatalog(fn func(CatalogEntry) error) error

func (*ChunkedArchiveReader) ReadEntryAt added in v0.14.0 

func (r *ChunkedArchiveReader) ReadEntryAt(offset int64) (*pxar.Entry, error)

func (*ChunkedArchiveReader) ReadEntryAtMinimal added in v0.14.0 

func (r *ChunkedArchiveReader) ReadEntryAtMinimal(offset int64) (*pxar.Entry, error)

func (*ChunkedArchiveReader) ReadFileContent 

func (r *ChunkedArchiveReader) ReadFileContent(entry *pxar.Entry) ([]byte, error)

func (*ChunkedArchiveReader) ReadFileContentReader added in v0.10.0 

func (r *ChunkedArchiveReader) ReadFileContentReader(entry *pxar.Entry) (io.ReadCloser, error)

func (*ChunkedArchiveReader) ReadRoot 

func (r *ChunkedArchiveReader) ReadRoot() (*pxar.Entry, error)

type ChunkedArchiveWriter 

type ChunkedArchiveWriter struct {
	IndexData    []byte
	ChunkResults []datastore.ChunkResult
	// contains filtered or unexported fields
}

ChunkedArchiveWriter writes an archive by encoding it to a buffer, then chunking and storing via a local ChunkStore, producing a .didx index.

func NewChunkedArchiveWriter 

func NewChunkedArchiveWriter(store *datastore.ChunkStore, config buzhash.Config, compress bool) *ChunkedArchiveWriter

NewChunkedArchiveWriter creates a chunked writer backed by a local chunk store.

func (*ChunkedArchiveWriter) Begin 

func (w *ChunkedArchiveWriter) Begin(rootMeta *pxar.Metadata, opts WriterOptions) error

func (*ChunkedArchiveWriter) BeginDirectory 

func (w *ChunkedArchiveWriter) BeginDirectory(name string, meta *pxar.Metadata) error

func (*ChunkedArchiveWriter) Close 

func (w *ChunkedArchiveWriter) Close() error

func (*ChunkedArchiveWriter) EndDirectory 

func (w *ChunkedArchiveWriter) EndDirectory() error

func (*ChunkedArchiveWriter) Finish 

func (w *ChunkedArchiveWriter) Finish() error

func (*ChunkedArchiveWriter) PayloadIndexData 

func (w *ChunkedArchiveWriter) PayloadIndexData() ([]byte, error)

PayloadIndexData returns the .didx index data for the payload stream. Only valid for v2 split archives after Finish.

func (*ChunkedArchiveWriter) WriteEntry 

func (w *ChunkedArchiveWriter) WriteEntry(entry *pxar.Entry, content []byte) error

func (*ChunkedArchiveWriter) WriteEntryReader added in v0.13.0 

func (w *ChunkedArchiveWriter) WriteEntryReader(entry *pxar.Entry, r io.Reader, size uint64) error

func (*ChunkedArchiveWriter) WriteEntryRef added in v0.17.0 

func (w *ChunkedArchiveWriter) WriteEntryRef(entry *pxar.Entry, payloadOffset uint64) error

type ChunkedReadSeeker 

type ChunkedReadSeeker struct {
	// contains filtered or unexported fields
}

ChunkedReadSeeker implements io.ReadSeeker over a chunked archive stream. Instead of reconstructing the entire stream into memory, it lazily loads and decodes chunks on demand using the dynamic index and a chunk source. This is critical for same-datastore transfers where only a subset of files are needed — it avoids downloading the entire payload stream from PBS.

func NewChunkedReadSeeker 

func NewChunkedReadSeeker(idx *datastore.DynamicIndexReader, source datastore.ChunkSource, maxCache int) *ChunkedReadSeeker

NewChunkedReadSeeker creates a lazy read-seeker over chunked data. maxCache controls how many decoded chunks are kept in memory (0 = unlimited).

func (*ChunkedReadSeeker) Close 

func (r *ChunkedReadSeeker) Close() error

Close clears the chunk cache.

func (*ChunkedReadSeeker) Read 

func (r *ChunkedReadSeeker) Read(p []byte) (int, error)

func (*ChunkedReadSeeker) ReadAt added in v0.16.0 

func (r *ChunkedReadSeeker) ReadAt(p []byte, offset int64) (int, error)

ReadAt reads len(p) bytes starting at the given offset without mutating the seeker's internal position. It is safe for concurrent use.

func (*ChunkedReadSeeker) Seek 

func (r *ChunkedReadSeeker) Seek(offset int64, whence int) (int64, error)

func (*ChunkedReadSeeker) SetCacheSize added in v0.16.7 

func (r *ChunkedReadSeeker) SetCacheSize(n int)

SetCacheSize adjusts the maximum number of decoded chunks kept in memory. Setting to 0 disables caching entirely — each chunk is decoded on demand and immediately discarded. This is appropriate for payload streams where content is streamed sequentially and caching would accumulate unbounded memory. Existing cached entries are evicted if the new size is lower.

type DecryptingChunkSource 

type DecryptingChunkSource struct {
	// contains filtered or unexported fields
}

DecryptingChunkSource wraps a ChunkSource and decrypts/decompresses chunks on the fly. This is used when reading from an encrypted archive where the raw chunks are encrypted blobs that need to be decoded before restoration.

When a CryptConfig is provided, encrypted blobs are decrypted. All blobs are decoded (uncompressed/decrypted) before being returned, producing the raw chunk data that the Restorer expects.

func NewDecryptingChunkSource 

func NewDecryptingChunkSource(inner datastore.ChunkSource, cc *datastore.CryptConfig) *DecryptingChunkSource

NewDecryptingChunkSource creates a chunk source that decrypts chunks after retrieval. Pass nil for cc if the archive is not encrypted (only decompression is needed).

func (*DecryptingChunkSource) GetChunk 

func (d *DecryptingChunkSource) GetChunk(digest [32]byte) ([]byte, error)

type DecryptingReader 

type DecryptingReader struct {
	// contains filtered or unexported fields
}

DecryptingReader is a placeholder for per-file decryption support. For most cases, using DecryptingChunkSource when constructing the underlying reader (ChunkedArchiveReader/SplitArchiveReader) is preferred since it handles decryption at the chunk level before stream reconstruction. This type simply delegates to the inner reader.

func NewDecryptingReader 

func NewDecryptingReader(inner ArchiveReader) *DecryptingReader

NewDecryptingReader wraps an ArchiveReader for transparent access. The underlying reader should already be configured with a DecryptingChunkSource if decryption is needed.

func (*DecryptingReader) Close 

func (r *DecryptingReader) Close() error

func (*DecryptingReader) ListDirectory 

func (r *DecryptingReader) ListDirectory(dirOffset int64, opts accessor.ListOption, fn func(*pxar.Entry) error) error

func (*DecryptingReader) Lookup 

func (r *DecryptingReader) Lookup(path string) (*pxar.Entry, error)

func (*DecryptingReader) ReadFileContent 

func (r *DecryptingReader) ReadFileContent(entry *pxar.Entry) ([]byte, error)

func (*DecryptingReader) ReadRoot 

func (r *DecryptingReader) ReadRoot() (*pxar.Entry, error)

type DedupSplitArchiveWriter 

type DedupSplitArchiveWriter struct {
	// contains filtered or unexported fields
}

DedupSplitArchiveWriter writes a v2 split archive into the same chunk store as the source, reusing existing payload chunks instead of re-uploading them.

For same-datastore transfers, this avoids:

  • Re-encoding file content into a new payload buffer
  • Re-chunking the payload stream (which would produce different chunks)
  • Re-uploading chunks that already exist in the store

Instead, it builds the new payload index by referencing source payload chunk digests directly. Only the metadata stream needs to be re-chunked and uploaded (it's small since it only contains filenames and metadata).

The writer assumes the source's payload chunks exist in the chunk store. For files not present in the source (new content), it falls back to normal encoding and chunking.

func NewDedupSplitArchiveWriter 

func NewDedupSplitArchiveWriter(
	store *datastore.ChunkStore,
	source datastore.ChunkSource,
	config buzhash.Config,
	compress bool,
	sourcePayloadIdx *datastore.DynamicIndexReader,
) *DedupSplitArchiveWriter

NewDedupSplitArchiveWriter creates a writer that reuses source payload chunks. sourcePayloadIdx is the source archive's .ppxar.didx index. source is the ChunkSource for reading source chunks (same store as target).

func (*DedupSplitArchiveWriter) Begin 

func (w *DedupSplitArchiveWriter) Begin(rootMeta *pxar.Metadata, opts WriterOptions) error

func (*DedupSplitArchiveWriter) BeginDirectory 

func (w *DedupSplitArchiveWriter) BeginDirectory(name string, meta *pxar.Metadata) error

func (*DedupSplitArchiveWriter) Close 

func (w *DedupSplitArchiveWriter) Close() error

func (*DedupSplitArchiveWriter) DedupStats 

func (w *DedupSplitArchiveWriter) DedupStats() (hits, total int)

DedupStats returns (already_existed, total) payload chunk counts.

func (*DedupSplitArchiveWriter) EndDirectory 

func (w *DedupSplitArchiveWriter) EndDirectory() error

func (*DedupSplitArchiveWriter) Finish 

func (w *DedupSplitArchiveWriter) Finish() error

func (*DedupSplitArchiveWriter) MetaIndexData 

func (w *DedupSplitArchiveWriter) MetaIndexData() []byte

MetaIndexData returns the .mpxar.didx index data after Finish.

func (*DedupSplitArchiveWriter) PayloadIndexData 

func (w *DedupSplitArchiveWriter) PayloadIndexData() []byte

PayloadIndexData returns the .ppxar.didx index data after Finish.

func (*DedupSplitArchiveWriter) ReferenceSourcePayloadChunks 

func (w *DedupSplitArchiveWriter) ReferenceSourcePayloadChunks()

ReferenceSourcePayloadChunks marks chunks from the source's payload index as already existing in the store. Call this before Finish to enable dedup tracking. The ChunkStore.InsertChunk call will skip these chunks since they already exist on disk.

This is mainly useful for reporting — the actual dedup happens automatically via ChunkStore.InsertChunk.

func (*DedupSplitArchiveWriter) WriteEntry 

func (w *DedupSplitArchiveWriter) WriteEntry(entry *pxar.Entry, content []byte) error

func (*DedupSplitArchiveWriter) WriteEntryReader added in v0.13.0 

func (w *DedupSplitArchiveWriter) WriteEntryReader(entry *pxar.Entry, r io.Reader, size uint64) error

type FileArchiveReader 

type FileArchiveReader struct {
	// contains filtered or unexported fields
}

FileArchiveReader reads from a standalone .pxar file using an io.ReadSeeker. For split archives (v2), provide both the metadata and payload readers.

func NewFileArchiveReader 

func NewFileArchiveReader(reader io.ReadSeeker) *FileArchiveReader

NewFileArchiveReader creates a reader for a standalone .pxar file.

func NewSplitFileArchiveReader 

func NewSplitFileArchiveReader(metaReader, payloadReader io.ReadSeeker) *FileArchiveReader

NewSplitFileArchiveReader creates a reader for a split (v2) archive with separate metadata and payload streams.

func (*FileArchiveReader) Close 

func (r *FileArchiveReader) Close() error

func (*FileArchiveReader) ListDirectory 

func (r *FileArchiveReader) ListDirectory(dirOffset int64, opts accessor.ListOption, fn func(*pxar.Entry) error) error

func (*FileArchiveReader) Lookup 

func (r *FileArchiveReader) Lookup(path string) (*pxar.Entry, error)

func (*FileArchiveReader) ReadCatalog added in v0.10.0 

func (r *FileArchiveReader) ReadCatalog(fn func(CatalogEntry) error) error

func (*FileArchiveReader) ReadEntryAt added in v0.14.0 

func (r *FileArchiveReader) ReadEntryAt(offset int64) (*pxar.Entry, error)

ReadEntryAt reads a full pxar entry at the given archive byte offset.

func (*FileArchiveReader) ReadEntryAtMinimal added in v0.14.0 

func (r *FileArchiveReader) ReadEntryAtMinimal(offset int64) (*pxar.Entry, error)

ReadEntryAtMinimal reads a pxar entry with minimal decoding (stat only).

func (*FileArchiveReader) ReadFileContent 

func (r *FileArchiveReader) ReadFileContent(entry *pxar.Entry) ([]byte, error)

func (*FileArchiveReader) ReadFileContentReader added in v0.10.0 

func (r *FileArchiveReader) ReadFileContentReader(entry *pxar.Entry) (io.ReadCloser, error)

func (*FileArchiveReader) ReadRoot 

func (r *FileArchiveReader) ReadRoot() (*pxar.Entry, error)

type MetaWalkFunc added in v0.9.0 

type MetaWalkFunc func(entry *pxar.Entry) error

MetaWalkFunc is called for each entry during a metadata-only walk. Unlike WalkFunc, no content parameter is provided since content is never read.

type PBSArchiveConfig 

type PBSArchiveConfig struct {
	BackupType  string
	BackupID    string
	ArchiveName string
	MetaName    string
	PayloadName string
	Config      backupproxy.PBSConfig
	BackupTime  int64
	MetaOnly    bool
}

PBSArchiveConfig holds the configuration for opening a PBS archive.

type PBSArchiveReader 

type PBSArchiveReader struct {
	// contains filtered or unexported fields
}

PBSArchiveReader reads archives from a PBS remote store. It downloads the index file(s) and reconstructs the archive stream using chunks from the PBS reader protocol.

func NewPBSArchiveReader 

func NewPBSArchiveReader(ctx context.Context, cfg PBSArchiveConfig) (*PBSArchiveReader, error)

NewPBSArchiveReader creates a reader for a PBS remote archive. For v1 archives, set ArchiveName. For v2 split archives, set MetaName and PayloadName.

func (*PBSArchiveReader) Close 

func (r *PBSArchiveReader) Close() error

func (*PBSArchiveReader) ListDirectory 

func (r *PBSArchiveReader) ListDirectory(dirOffset int64, opts accessor.ListOption, fn func(*pxar.Entry) error) error

func (*PBSArchiveReader) Lookup 

func (r *PBSArchiveReader) Lookup(path string) (*pxar.Entry, error)

func (*PBSArchiveReader) ReadCatalog added in v0.10.0 

func (r *PBSArchiveReader) ReadCatalog(fn func(CatalogEntry) error) error

func (*PBSArchiveReader) ReadFileContent 

func (r *PBSArchiveReader) ReadFileContent(entry *pxar.Entry) ([]byte, error)

func (*PBSArchiveReader) ReadFileContentReader added in v0.10.0 

func (r *PBSArchiveReader) ReadFileContentReader(entry *pxar.Entry) (io.ReadCloser, error)

func (*PBSArchiveReader) ReadRoot 

func (r *PBSArchiveReader) ReadRoot() (*pxar.Entry, error)

type PathMapping 

type PathMapping struct {
	Src string // path in the source archive
	Dst string // path in the target archive
}

PathMapping maps a source path to a destination path inside the archives.

type RemoteDedupSplitArchiveWriter added in v0.17.0 

type RemoteDedupSplitArchiveWriter struct {
	// contains filtered or unexported fields
}

RemoteDedupSplitArchiveWriter writes a split archive to PBS with chunk-level dedup.

For unchanged files (pxar-only entries), it uses AddPayloadRef to reference original payload offsets without reading file content. The original payload chunks are injected into the new DIDX by reference.

For new/modified files (backed entries), it writes payload data normally.

Only metadata and new file content are buffered in memory. Unchanged file content is never read — the original payload chunks are referenced directly.

func NewRemoteDedupSplitArchiveWriter added in v0.17.0 

func NewRemoteDedupSplitArchiveWriter(
	ctx context.Context,
	session backupproxy.BackupSession,
	metaName, payloadName string,
	origPayloadIndex []byte,
) (*RemoteDedupSplitArchiveWriter, error)

NewRemoteDedupSplitArchiveWriter creates a dedup writer for PBS uploads. origPayloadIndex is the raw DIDX bytes from the original .ppxar.didx.

func (*RemoteDedupSplitArchiveWriter) Begin added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) Begin(rootMeta *pxar.Metadata, opts WriterOptions) error

func (*RemoteDedupSplitArchiveWriter) BeginDirectory added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) BeginDirectory(name string, meta *pxar.Metadata) error

func (*RemoteDedupSplitArchiveWriter) Close added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) Close() error

func (*RemoteDedupSplitArchiveWriter) Encoder added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) Encoder() *encoder.Encoder

Encoder returns the underlying encoder.

func (*RemoteDedupSplitArchiveWriter) EndDirectory added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) EndDirectory() error

func (*RemoteDedupSplitArchiveWriter) Finish added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) Finish() error

func (*RemoteDedupSplitArchiveWriter) WriteEntry added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) WriteEntry(entry *pxar.Entry, content []byte) error

func (*RemoteDedupSplitArchiveWriter) WriteEntryReader added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) WriteEntryReader(entry *pxar.Entry, r io.Reader, size uint64) error

func (*RemoteDedupSplitArchiveWriter) WriteEntryRef added in v0.17.0 

func (w *RemoteDedupSplitArchiveWriter) WriteEntryRef(entry *pxar.Entry, payloadOffset uint64) error

WriteEntryRef writes an entry referencing existing payload data.

type SessionArchiveWriter 

type SessionArchiveWriter struct {
	UploadResult *backupproxy.UploadResult
	// contains filtered or unexported fields
}

SessionArchiveWriter writes an archive by uploading it through a BackupSession. This works with both local stores and PBS remote stores.

func NewSessionArchiveWriter 

func NewSessionArchiveWriter(ctx context.Context, session backupproxy.BackupSession, name string) *SessionArchiveWriter

NewSessionArchiveWriter creates a writer that uploads via a BackupSession.

func (*SessionArchiveWriter) Begin 

func (w *SessionArchiveWriter) Begin(rootMeta *pxar.Metadata, opts WriterOptions) error

func (*SessionArchiveWriter) BeginDirectory 

func (w *SessionArchiveWriter) BeginDirectory(name string, meta *pxar.Metadata) error

func (*SessionArchiveWriter) Close 

func (w *SessionArchiveWriter) Close() error

func (*SessionArchiveWriter) EndDirectory 

func (w *SessionArchiveWriter) EndDirectory() error

func (*SessionArchiveWriter) Finish 

func (w *SessionArchiveWriter) Finish() error

func (*SessionArchiveWriter) WriteEntry 

func (w *SessionArchiveWriter) WriteEntry(entry *pxar.Entry, content []byte) error

func (*SessionArchiveWriter) WriteEntryReader added in v0.13.0 

func (w *SessionArchiveWriter) WriteEntryReader(entry *pxar.Entry, r io.Reader, size uint64) error

func (*SessionArchiveWriter) WriteEntryRef added in v0.17.0 

func (w *SessionArchiveWriter) WriteEntryRef(entry *pxar.Entry, payloadOffset uint64) error

type SplitArchiveReader 

type SplitArchiveReader struct {
	// contains filtered or unexported fields
}

SplitArchiveReader reads from a split chunked archive (.mpxar.didx + .ppxar.didx). It uses lazy on-demand chunk loading for both metadata and payload streams, avoiding full-stream-in-memory reconstruction. For small archives, use NewSplitArchiveReaderEager.

func NewSplitArchiveReader 

func NewSplitArchiveReader(metaIdxData, payloadIdxData []byte, source datastore.ChunkSource) (*SplitArchiveReader, error)

NewSplitArchiveReader creates a reader for a split chunked archive using lazy on-demand chunk loading. Only chunks needed for Lookups and ReadFileContent calls are loaded, which is critical for same-datastore PBS transfers where downloading the entire payload stream is expensive.

func NewSplitArchiveReaderEager 

func NewSplitArchiveReaderEager(metaIdxData, payloadIdxData []byte, source datastore.ChunkSource) (*SplitArchiveReader, error)

NewSplitArchiveReaderEager creates a reader that reconstructs both streams into memory upfront. Use for small archives or when you need guaranteed sequential access performance.

func NewSplitArchiveReaderMetaOnly added in v0.10.0 

func NewSplitArchiveReaderMetaOnly(metaIdxData []byte, source datastore.ChunkSource) (*SplitArchiveReader, error)

NewSplitArchiveReaderMetaOnly creates a reader for a split archive that only downloads and uses the metadata stream. The payload stream is never fetched. ReadFileContent/ReadFileContentReader will return errors for files stored in the payload stream (PayloadOffset > 0).

func (*SplitArchiveReader) Close 

func (r *SplitArchiveReader) Close() error

func (*SplitArchiveReader) ListDirectory 

func (r *SplitArchiveReader) ListDirectory(dirOffset int64, opts accessor.ListOption, fn func(*pxar.Entry) error) error

func (*SplitArchiveReader) Lookup 

func (r *SplitArchiveReader) Lookup(path string) (*pxar.Entry, error)

func (*SplitArchiveReader) ReadCatalog added in v0.10.0 

func (r *SplitArchiveReader) ReadCatalog(fn func(CatalogEntry) error) error

func (*SplitArchiveReader) ReadEntryAt added in v0.14.0 

func (r *SplitArchiveReader) ReadEntryAt(offset int64) (*pxar.Entry, error)

func (*SplitArchiveReader) ReadEntryAtMinimal added in v0.14.0 

func (r *SplitArchiveReader) ReadEntryAtMinimal(offset int64) (*pxar.Entry, error)

func (*SplitArchiveReader) ReadFileContent 

func (r *SplitArchiveReader) ReadFileContent(entry *pxar.Entry) ([]byte, error)

func (*SplitArchiveReader) ReadFileContentReader added in v0.10.0 

func (r *SplitArchiveReader) ReadFileContentReader(entry *pxar.Entry) (io.ReadCloser, error)

func (*SplitArchiveReader) ReadRoot 

func (r *SplitArchiveReader) ReadRoot() (*pxar.Entry, error)

func (*SplitArchiveReader) SetPayloadCacheSize added in v0.16.7 

func (r *SplitArchiveReader) SetPayloadCacheSize(n int)

SetPayloadCacheSize adjusts the payload chunk cache size. See ChunkedReadSeeker.SetCacheSize for details.

type SplitSessionArchiveWriter 

type SplitSessionArchiveWriter struct {
	SplitResult *backupproxy.SplitArchiveResult
	// contains filtered or unexported fields
}

SplitSessionArchiveWriter writes a split (v2) archive by uploading both metadata and payload streams through a BackupSession.

func NewSplitSessionArchiveWriter 

func NewSplitSessionArchiveWriter(ctx context.Context, session backupproxy.BackupSession, metaName, payloadName string) *SplitSessionArchiveWriter

NewSplitSessionArchiveWriter creates a split writer that uploads via a BackupSession.

func (*SplitSessionArchiveWriter) Begin 

func (w *SplitSessionArchiveWriter) Begin(rootMeta *pxar.Metadata, opts WriterOptions) error

func (*SplitSessionArchiveWriter) BeginDirectory 

func (w *SplitSessionArchiveWriter) BeginDirectory(name string, meta *pxar.Metadata) error

func (*SplitSessionArchiveWriter) Close 

func (w *SplitSessionArchiveWriter) Close() error

func (*SplitSessionArchiveWriter) Encoder added in v0.17.0 

func (w *SplitSessionArchiveWriter) Encoder() *encoder.Encoder

Encoder returns the underlying encoder for advanced operations.

func (*SplitSessionArchiveWriter) EndDirectory 

func (w *SplitSessionArchiveWriter) EndDirectory() error

func (*SplitSessionArchiveWriter) Finish 

func (w *SplitSessionArchiveWriter) Finish() error

func (*SplitSessionArchiveWriter) WriteEntry 

func (w *SplitSessionArchiveWriter) WriteEntry(entry *pxar.Entry, content []byte) error

func (*SplitSessionArchiveWriter) WriteEntryReader added in v0.13.0 

func (w *SplitSessionArchiveWriter) WriteEntryReader(entry *pxar.Entry, r io.Reader, size uint64) error

func (*SplitSessionArchiveWriter) WriteEntryRef added in v0.17.0 

func (w *SplitSessionArchiveWriter) WriteEntryRef(entry *pxar.Entry, payloadOffset uint64) error

type StreamArchiveWriter 

type StreamArchiveWriter struct {
	// contains filtered or unexported fields
}

StreamArchiveWriter writes a pxar archive to one or two io.Writer streams. For v1 format, only output is used. For v2 format, both output and payloadOut are used.

func NewSplitStreamArchiveWriter 

func NewSplitStreamArchiveWriter(output, payloadOut io.Writer) *StreamArchiveWriter

NewSplitStreamArchiveWriter creates a writer for v2 (split) format.

func NewStreamArchiveWriter 

func NewStreamArchiveWriter(output io.Writer) *StreamArchiveWriter

NewStreamArchiveWriter creates a writer for v1 (unified) format.

func (*StreamArchiveWriter) Begin 

func (w *StreamArchiveWriter) Begin(rootMeta *pxar.Metadata, opts WriterOptions) error

func (*StreamArchiveWriter) BeginDirectory 

func (w *StreamArchiveWriter) BeginDirectory(name string, meta *pxar.Metadata) error

func (*StreamArchiveWriter) Close 

func (w *StreamArchiveWriter) Close() error

func (*StreamArchiveWriter) Encoder 

func (w *StreamArchiveWriter) Encoder() *encoder.Encoder

Encoder returns the underlying encoder for advanced operations. This is useful for getting file offsets for hardlink tracking.

func (*StreamArchiveWriter) EndDirectory 

func (w *StreamArchiveWriter) EndDirectory() error

func (*StreamArchiveWriter) Finish 

func (w *StreamArchiveWriter) Finish() error

func (*StreamArchiveWriter) WriteEntry 

func (w *StreamArchiveWriter) WriteEntry(entry *pxar.Entry, content []byte) error

func (*StreamArchiveWriter) WriteEntryReader added in v0.13.0 

func (w *StreamArchiveWriter) WriteEntryReader(entry *pxar.Entry, r io.Reader, size uint64) error

func (*StreamArchiveWriter) WriteEntryRef added in v0.17.0 

func (w *StreamArchiveWriter) WriteEntryRef(entry *pxar.Entry, payloadOffset uint64) error
func (w *StreamArchiveWriter) WriteHardlink(name string, target string, targetOffset encoder.LinkOffset) error

WriteHardlink writes a hard link entry with an explicit target offset.

type TransferOption 

type TransferOption struct {
	SourceCryptConfig *datastore.CryptConfig
	TargetCryptConfig *datastore.CryptConfig
	OnProgress        func(path string, bytes uint64)
	TargetFormat      format.FormatVersion
	Overwrite         bool
}

TransferOption configures a file transfer operation.

type TreeWalker added in v0.9.0 

type TreeWalker struct {
	// contains filtered or unexported fields
}

TreeWalker provides a pull-based iterator for walking a pxar archive tree. It reuses a single Entry across all Next() calls, producing zero heap allocations per iteration.

Example: 

walker := transfer.NewTreeWalker(reader, transfer.WalkOption{
    MetaOnly: true,
    Filter:   transfer.WalkFiles | transfer.WalkDirs,
})
if err := walker.Init("/"); err != nil { ... }
for walker.Next() {
    entry := walker.Entry()
    // entry is reused each iteration — copy values you need to keep
}
if err := walker.Err(); err != nil { ... }

func NewTreeWalker added in v0.9.0 

func NewTreeWalker(reader ArchiveReader, opts WalkOption) *TreeWalker

NewTreeWalker creates a pull-based walker for the archive. Call Init to set the root path before calling Next.

func (*TreeWalker) Entry added in v0.9.0 

func (w *TreeWalker) Entry() *pxar.Entry

Entry returns the current entry. The returned pointer is valid only until the next call to Next. The same Entry memory is reused each iteration.

func (*TreeWalker) Err added in v0.9.0 

func (w *TreeWalker) Err() error

Err returns the error that stopped iteration, if any.

func (*TreeWalker) Init added in v0.9.0 

func (w *TreeWalker) Init(rootPath string) error

Init resolves the root entry and prepares the walker for iteration. Must be called before Next.

func (*TreeWalker) Next added in v0.9.0 

func (w *TreeWalker) Next() bool

Next advances to the next entry matching the walk filter. Returns false when there are no more entries or on error (check Err).

type WalkFilter added in v0.9.0 

type WalkFilter uint

WalkFilter is a bitmask that controls which entry types are visited during a walk. Entries whose type is not in the mask are skipped entirely — the callback is never invoked for them, and directories are not descended into. 

const (
	WalkFiles     WalkFilter = 1 << iota // regular files
	WalkDirs                             // directories
	WalkSymlinks                         // symbolic links
	WalkHardlinks                        // hard links
	WalkDevices                          // device nodes
	WalkFifos                            // named pipes (FIFOs)
	WalkSockets                          // unix sockets

	WalkAll WalkFilter = WalkFiles | WalkDirs | WalkSymlinks |
		WalkHardlinks | WalkDevices | WalkFifos | WalkSockets
)

type WalkFunc 

type WalkFunc func(entry *pxar.Entry, content []byte) error

WalkFunc is called for each entry encountered during WalkTree. entry is the archive entry. content is the file data (nil for non-files). Return nil to continue, or an error to stop.

type WalkOption added in v0.9.0 

type WalkOption struct {
	// MetaOnly skips reading file content. When true, content is never read
	// from the archive and the content parameter passed to WalkFunc is always nil.
	MetaOnly bool

	// Filter is a bitmask of entry types to include. Entries not matching
	// the filter are skipped without invoking the callback. Directories that
	// are filtered out are not descended into. Zero means accept all types.
	Filter WalkFilter

	// SkipCount fast-forwards past the first N entries without invoking the
	// callback. Entries are still decoded but the walk callback is skipped.
	// Useful for resuming a previous walk.
	SkipCount int
}

WalkOption configures walk behavior. The zero value walks all entry types and reads file content (equivalent to the original WalkTree behavior).

type WriterOptions 

type WriterOptions struct {
	Prelude []byte
	Format  format.FormatVersion
}

WriterOptions configures how an ArchiveWriter creates archives.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.