backupproxy

package
v0.28.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 15, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package backupproxy provides a pull-mode backup architecture where a server (running on the PBS machine) orchestrates backups by walking a remote client's filesystem, encoding pxar archives, chunking with buzhash, and uploading to a backup store. The client only serves raw filesystem data.

The transport between server and client is pluggable — this package defines interfaces and message types. Users provide their own transport implementation (gRPC, HTTP, SSH, etc.).

Detection Modes

Three modes control how archives are created and whether unchanged files are re-read:

  • DetectionLegacy: v1 single .pxar, all data encoded in one stream
  • DetectionData: v2 split .mpxar + .ppxar, all data re-read
  • DetectionMetadata: v2 split, compares metadata against previous backup, reuses unchanged payload chunks

Encryption

Three crypt modes are supported:

  • CryptModeNone: no encryption or signing (default)
  • CryptModeEncrypt: AES-256-GCM encryption + HMAC-SHA256 manifest signing
  • CryptModeSignOnly: no encryption, HMAC-SHA256 manifest signing

Encryption uses PBKDF2-HMAC-SHA256 for key derivation and AES-256-GCM for chunk encryption. Manifests are signed but never encrypted (PBS must read them).

Extended Metadata

Extended attributes, POSIX ACLs, and file capabilities are collected via the FSAccessor/ClientProvider interfaces and encoded into archives. Metadata change detection compares all fields (stat, xattrs, ACLs, fcaps) to trigger re-upload when they change.

Backup Catalogs

All backup modes automatically generate and upload a catalog.pcat1.didx file, enabling PBS's web UI to browse backup contents without downloading the full archive.

PBS Reader Protocol

PBSReader provides access to the Proxmox Backup Server reader protocol (proxmox-backup-reader-protocol-v1) via HTTP/2. This enables downloading index files and individual chunks, and restoring files: 

reader := backupproxy.NewPBSReader(cfg, "host", "mybackup", backupTime)
reader.Connect(ctx)
defer reader.Close()

didxData, _ := reader.DownloadFile("root.pxar.didx")
idx, _ := datastore.ParseDynamicIndex(didxData)

var buf bytes.Buffer
reader.RestoreFile(idx, &buf)
// Or partial range:
reader.RestoreFileRange(idx, 1024, 1024, &buf)

Use AsChunkSource() to integrate with Restorer, ReadSeeker, etc.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EntryMatches added in v0.5.0 

func EntryMatches(current DirEntry, catalogMetadata pxar.Metadata, prev *SnapshotEntry) bool

EntryMatches checks if a current directory entry matches a catalog entry for metadata change detection. Returns true if the file metadata hasn't changed. Compares stat fields, file type, size, xattrs, ACLs, FCaps, and QuotaProjectID.

Types

type BackupConfig 

type BackupConfig struct {
	PreviousBackup *PreviousBackupRef
	CryptConfig    *datastore.CryptConfig
	BackupID       string
	Namespace      string
	CryptMode      datastore.CryptMode
	ChunkConfig    buzhash.Config
	BackupType     datastore.BackupType
	BackupTime     int64
	DetectionMode  DetectionMode
	Compress       bool
}

BackupConfig holds parameters for a single backup operation.

type BackupResult 

type BackupResult struct {
	Manifest        *datastore.Manifest
	TotalBytes      int64
	FileCount       int
	DirCount        int
	Duration        time.Duration
	CatalogUploaded bool
}

BackupResult describes the outcome of a backup operation.

type BackupSession 

type BackupSession interface {
	UploadArchive(ctx context.Context, name string, data io.Reader) (*UploadResult, error)
	UploadSplitArchive(ctx context.Context, metadataName string, metadataData io.Reader, payloadName string, payloadData io.Reader) (*SplitArchiveResult, error)
	UploadBlob(ctx context.Context, name string, data []byte) error
	UploadPayloadInterleaved(ctx context.Context, name string, newData io.Reader, injections <-chan InjectChunks) (*UploadResult, error)
	Finish(ctx context.Context) (*datastore.Manifest, error)
}

type ClientProvider 

type ClientProvider interface {
	Stat(ctx context.Context, path string) (format.Stat, error)
	ReadDir(ctx context.Context, path string) ([]DirEntry, error)
	// OpenFile returns a reader for streaming file content. The caller must
	// close the reader. The returned size is the total file size.
	OpenFile(ctx context.Context, path string) (io.ReadCloser, uint64, error)
	ReadLink(ctx context.Context, path string) (string, error)

	// Extended metadata methods for full archive fidelity.
	GetXAttrs(ctx context.Context, path string) ([]format.XAttr, error)
	GetACL(ctx context.Context, path string) (pxar.ACL, error)
	GetFCaps(ctx context.Context, path string) ([]byte, error)
}

ClientProvider is the interface the server uses to access client data. Transport implementations bridge this to the actual network. Each method call corresponds to one request-response round trip.

type DetectionMode added in v0.5.0 

type DetectionMode int

DetectionMode controls how file changes are detected between backup runs. 

const (
	// DetectionLegacy creates a single self-contained pxar v1 archive.
	// All file data and metadata are read and encoded in one stream.
	DetectionLegacy DetectionMode = iota

	// DetectionData creates split pxar v2 archives (.mpxar + .ppxar).
	// All file data is still read fully, but metadata and payload are
	// stored in separate streams for more efficient catalog access.
	DetectionData

	// DetectionMetadata creates split pxar v2 archives (.mpxar + .ppxar)
	// but only re-reads file content for files whose metadata (size, mtime,
	// uid, gid, mode, xattrs) has changed since the previous backup.
	// Unchanged files reuse payload chunks from the previous backup.
	DetectionMetadata
)

func (DetectionMode) String added in v0.5.0 

func (d DetectionMode) String() string

type DirEntry 

type DirEntry struct {
	QuotaProjectID *uint64
	Name           string
	ACL            pxar.ACL
	XAttrs         []format.XAttr
	FCaps          []byte
	Stat           format.Stat
	Size           uint64
}

DirEntry represents a single entry from a directory listing on the client.

type FSAccessor added in v0.24.0 

type FSAccessor interface {
	Stat(path string) (format.Stat, error)
	ReadDir(path string) ([]DirEntry, error)
	// OpenFile returns a reader for streaming file content. The caller must
	// close the reader. The returned size is the total file size.
	OpenFile(path string) (io.ReadCloser, uint64, error)
	ReadLink(path string) (string, error)

	// GetXAttrs returns the extended attributes for the given path.
	GetXAttrs(path string) ([]format.XAttr, error)

	// GetACL returns the POSIX ACL entries for the given path.
	GetACL(path string) (pxar.ACL, error)

	// GetFCaps returns the file capabilities for the given path.
	GetFCaps(path string) ([]byte, error)
}

FSAccessor is the interface the client uses to access the local filesystem. Users provide their own implementation. This indirection allows testing without a real filesystem and running in constrained environments.

type FileOpener added in v0.18.0 

type FileOpener interface {
	// OpenFile returns a reader for the file at the given path. The caller
	// must close the reader. The returned size is the total file size.
	OpenFile(ctx context.Context, path string) (io.ReadCloser, uint64, error)
}

FileOpener is an optional interface for streaming file reads. Clients that implement this allow the server to stream file content directly into the archive encoder without buffering the entire file.

type InjectChunks added in v0.28.0 

type InjectChunks struct {
	Chunks   []KnownChunkRef
	Size     uint64
	Boundary uint64
}

InjectChunks describes a batch of reused chunks to inject into the payload stream at a specific offset.

Boundary is the absolute payload-stream offset (encoder.PayloadPosition) at which the injection occurs, before the encoder is advanced by Size. It mirrors the `boundary` field of pbs-client's InjectChunks and is what the payload chunker uses to interleave injected chunks with new data in the correct offset order. Without it the new-data and injected-chunk offsets drift apart, producing server errors like "strange chunk offset".

type KnownChunkRef added in v0.17.0 

type KnownChunkRef struct {
	Digest [32]byte
	Size   uint64 // chunk size in bytes
}

BackupSession represents an active backup upload session. KnownChunkRef references a chunk already stored in the datastore.

type LocalClient 

type LocalClient struct {
	// contains filtered or unexported fields
}

LocalClient implements ClientProvider by delegating to a FSAccessor. This is the client-side component: runs on the machine being backed up.

func NewLocalClient 

func NewLocalClient(fs FSAccessor) *LocalClient

NewLocalClient creates a client backed by the given FSAccessor.

func (*LocalClient) GetACL added in v0.6.0 

func (lc *LocalClient) GetACL(_ context.Context, path string) (pxar.ACL, error)

GetACL returns the POSIX ACL entries for the given path.

func (*LocalClient) GetFCaps added in v0.6.0 

func (lc *LocalClient) GetFCaps(_ context.Context, path string) ([]byte, error)

GetFCaps returns the file capabilities for the given path.

func (*LocalClient) GetXAttrs added in v0.6.0 

func (lc *LocalClient) GetXAttrs(_ context.Context, path string) ([]format.XAttr, error)

GetXAttrs returns the extended attributes for the given path.

func (*LocalClient) OpenFile added in v0.18.0 

func (lc *LocalClient) OpenFile(_ context.Context, path string) (io.ReadCloser, uint64, error)

OpenFile delegates to the underlying FSAccessor.

func (*LocalClient) ReadDir 

func (lc *LocalClient) ReadDir(_ context.Context, path string) ([]DirEntry, error)

ReadDir returns directory entries for the given path by delegating to the underlying FSAccessor. 

func (lc *LocalClient) ReadLink(_ context.Context, path string) (string, error)

ReadLink returns the symlink target for the given path by delegating to the underlying FSAccessor.

func (*LocalClient) Stat 

func (lc *LocalClient) Stat(_ context.Context, path string) (format.Stat, error)

Stat returns file metadata for the given path by delegating to the underlying FSAccessor.

type LocalStore 

type LocalStore struct {
	// contains filtered or unexported fields
}

LocalStore implements RemoteStore using a local filesystem directory. It uses datastore.ChunkStore for chunk storage and writes index/blob files to disk. Intended for testing and offline backups.

func NewLocalStore 

func NewLocalStore(baseDir string, config buzhash.Config, compress bool) (*LocalStore, error)

NewLocalStore creates a LocalStore backed by the given directory.

func (*LocalStore) NewPreviousSnapshotSource added in v0.5.0 

func (ls *LocalStore) NewPreviousSnapshotSource(_ context.Context, _ datastore.BackupType, _ string, _ int64, _ string) (PreviousSnapshotSource, error)

NewPreviousSnapshotSource creates a PreviousSnapshotSource for a local backup snapshot.

func (*LocalStore) ReadPreviousArchive added in v0.5.0 

func (ls *LocalStore) ReadPreviousArchive(_ context.Context, _ datastore.BackupType, _ string, _ int64, _, filename string) ([]byte, error)

ReadPreviousArchive reads an archive file from a previous local backup snapshot.

func (*LocalStore) StartSession 

func (ls *LocalStore) StartSession(_ context.Context, config BackupConfig) (BackupSession, error)

StartSession creates a new local backup session.

type NoExtendedAttrs added in v0.6.0 

type NoExtendedAttrs struct{}

NoExtendedAttrs provides no-op implementations of the extended metadata methods. Embed this in FSAccessor implementations that don't support xattrs, ACLs, or file capabilities.

func (NoExtendedAttrs) GetACL added in v0.6.0 

func (NoExtendedAttrs) GetACL(string) (pxar.ACL, error)

func (NoExtendedAttrs) GetFCaps added in v0.6.0 

func (NoExtendedAttrs) GetFCaps(string) ([]byte, error)

func (NoExtendedAttrs) GetXAttrs added in v0.6.0 

func (NoExtendedAttrs) GetXAttrs(string) ([]format.XAttr, error)

type PBSConfig 

type PBSConfig struct {
	BaseURL       string
	Datastore     string
	AuthToken     string
	Namespace     string
	SkipTLSVerify bool
}

PBSConfig holds configuration for connecting to a Proxmox Backup Server.

type PBSReader 

type PBSReader struct {
	// contains filtered or unexported fields
}

PBSReader provides read access to a PBS datastore via the reader protocol.

func NewPBSReader 

func NewPBSReader(config PBSConfig, backupType, backupID string, backupTime int64) *PBSReader

NewPBSReader creates a new PBS reader for the given backup snapshot.

func (*PBSReader) AsChunkSource 

func (r *PBSReader) AsChunkSource() datastore.ChunkSource

AsChunkSource returns a goroutine-safe ChunkSource for the restorer. The returned source serializes H2 encoder/framer access via an internal mutex, making MaxWorkers > 1 safe without external synchronization.

func (*PBSReader) Close 

func (r *PBSReader) Close() error

Close closes the reader connection.

func (*PBSReader) Connect 

func (r *PBSReader) Connect(ctx context.Context) error

Connect establishes the H2 reader connection to PBS.

func (*PBSReader) DownloadChunk 

func (r *PBSReader) DownloadChunk(digest [32]byte) ([]byte, error)

DownloadChunk downloads a chunk by its digest. The reader protocol requires that the index file referencing this chunk has been downloaded first (via DownloadFile), which populates the server-side allowed_chunks set.

func (*PBSReader) DownloadFile 

func (r *PBSReader) DownloadFile(fileName string) ([]byte, error)

DownloadFile downloads an index file (.didx, .fidx, .blob) from PBS.

func (*PBSReader) RestoreFile 

func (r *PBSReader) RestoreFile(idx *datastore.DynamicIndexReader, w io.Writer) error

RestoreFile restores a complete file from a dynamic index. This downloads all chunks and reconstructs the file content. The index file must have been downloaded first (via DownloadFile) to populate the server-side allowed_chunks set.

func (*PBSReader) RestoreFileRange 

func (r *PBSReader) RestoreFileRange(idx *datastore.DynamicIndexReader, offset, length uint64, w io.Writer) error

RestoreFileRange restores a specific byte range from a file. Useful for partial reads without downloading the entire file.

type PBSStore added in v0.18.0 

type PBSStore struct {
	// contains filtered or unexported fields
}

PBSStore implements RemoteStore via the PBS H2 backup protocol.

func NewPBSStore added in v0.18.0 

func NewPBSStore(config PBSConfig, chunkCfg buzhash.Config, compress bool) *PBSStore

NewPBSStore creates a PBS remote store with the given configuration.

func (*PBSStore) NewPreviousSnapshotSource added in v0.18.0 

func (ps *PBSStore) NewPreviousSnapshotSource(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace string) (PreviousSnapshotSource, error)

NewPreviousSnapshotSource creates a PreviousSnapshotSource connected to a PBS snapshot.

func (*PBSStore) ReadPreviousArchive added in v0.18.0 

func (ps *PBSStore) ReadPreviousArchive(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace, filename string) ([]byte, error)

ReadPreviousArchive reads an archive file from a previous PBS backup snapshot.

func (*PBSStore) StartSession added in v0.18.0 

func (ps *PBSStore) StartSession(ctx context.Context, config BackupConfig) (BackupSession, error)

StartSession dials PBS via H2 upgrade and returns a backup session.

type PreviousBackupRef added in v0.5.0 

type PreviousBackupRef struct {
	BackupID   string
	Namespace  string
	Dir        string
	BackupType datastore.BackupType
	BackupTime int64
}

PreviousBackupRef identifies a previous backup snapshot for metadata comparison.

type PreviousSnapshotSource added in v0.5.0 

type PreviousSnapshotSource interface {
	ReadArchive(filename string) ([]byte, error)
	ChunkSource() datastore.ChunkSource
	Close() error
}

PreviousSnapshotSource provides read access to a previous backup snapshot for metadata change detection. It can read archive files and download chunks.

func NewPreviousSnapshotSourceFromDir added in v0.5.0 

func NewPreviousSnapshotSourceFromDir(dir string) (PreviousSnapshotSource, error)

NewPreviousSnapshotSourceFromDir creates a PreviousSnapshotSource from a local directory.

type RemoteStore 

type RemoteStore interface {
	RemoteStoreBase
	SnapshotReader
}

RemoteStore abstracts the backup storage backend.

type RemoteStoreBase added in v0.5.0 

type RemoteStoreBase interface {
	StartSession(ctx context.Context, config BackupConfig) (BackupSession, error)
}

RemoteStoreBase contains the session creation method.

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server orchestrates pull backups: walks the client filesystem, encodes a pxar archive, chunks it with buzhash, and uploads to a RemoteStore.

func NewServer 

func NewServer(client ClientProvider, store RemoteStore) *Server

NewServer creates a backup server with the given client provider and store.

func (*Server) RunBackup 

func (s *Server) RunBackup(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunBackup executes a full pull backup using the legacy v1 format (single archive).

func (*Server) RunBackupWithMode added in v0.5.0 

func (s *Server) RunBackupWithMode(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunBackupWithMode dispatches to the appropriate backup method based on config.DetectionMode. It uses RunBackup for legacy, RunSplitBackup for data, and RunMetadataBackup for metadata mode.

func (*Server) RunMetadataBackup added in v0.5.0 

func (s *Server) RunMetadataBackup(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunMetadataBackup executes an incremental pull backup using metadata change detection. It downloads the previous backup's metadata and payload catalogs, compares current file metadata against them, and only reads content from the client for files that changed. Unchanged files reuse their payload data from the previous backup.

func (*Server) RunSplitBackup added in v0.4.0 

func (s *Server) RunSplitBackup(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunSplitBackup executes a full pull backup using the split archive format (v2, data mode). The encoder writes metadata and payload to buffers first, then uploads them sequentially via UploadSplitArchive. This avoids the io.Pipe deadlock that occurs when UploadSplitArchive reads one stream at a time while the encoder writes both simultaneously.

type SnapshotCatalog added in v0.13.0 

type SnapshotCatalog map[string]*SnapshotEntry

SnapshotCatalog is a map from normalized file path to SnapshotEntry.

func BuildCatalog added in v0.5.0 

func BuildCatalog(metaIdx *datastore.DynamicIndexReader, chunkSource datastore.ChunkSource) (SnapshotCatalog, error)

BuildCatalog constructs a metadata catalog from a previous backup's .mpxar.didx and .ppxar.didx indexes. It restores the metadata stream from chunks and walks it with the accessor to extract file entries.

type SnapshotEntry added in v0.13.0 

type SnapshotEntry struct {
	Path          string
	Metadata      pxar.Metadata
	Stat          format.Stat
	FileSize      uint64
	PayloadOffset uint64
	IsRegularFile bool
}

SnapshotEntry holds metadata from a previous backup's .mpxar archive, used for metadata change detection.

type SnapshotReader added in v0.5.0 

type SnapshotReader interface {
	ReadPreviousArchive(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace, filename string) ([]byte, error)
	NewPreviousSnapshotSource(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace string) (PreviousSnapshotSource, error)
}

SnapshotReader can read files from previous snapshots.

type SplitArchiveResult added in v0.4.0 

type SplitArchiveResult struct {
	MetadataResult *UploadResult
	PayloadResult  *UploadResult
}

SplitArchiveResult contains the results of uploading a split archive. The metadata and payload are uploaded as separate .didx files.

type UploadResult 

type UploadResult struct {
	Filename string   // e.g., "root.pxar.didx"
	Size     uint64   // total index size
	Digest   [32]byte // SHA-256 of the index
}

UploadResult describes the outcome of an archive upload.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.