backupproxy

package
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 14, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package backupproxy provides a pull-mode backup architecture where a server (running on the PBS machine) orchestrates backups by walking a remote client's filesystem, encoding pxar archives, chunking with buzhash, and uploading to a backup store. The client only serves raw filesystem data.

The transport between server and client is pluggable — this package defines interfaces and message types. Users provide their own transport implementation (gRPC, HTTP, SSH, etc.).

Features

All backup modes automatically generate and upload a catalog.pcat1.didx file, enabling PBS's web UI to browse backup contents without downloading the full archive.

The library supports three crypt modes:

  • CryptModeNone: no encryption or signing (default)
  • CryptModeEncrypt: AES-256-GCM encryption of chunk data; HMAC-SHA256 manifest signing
  • CryptModeSign: no encryption, but HMAC-SHA256 manifest signing for integrity verification

Encryption uses PBKDF2-HMAC-SHA256 for key derivation and AES-256-GCM for chunk encryption. Manifests are signed but never encrypted (PBS must read them).

Extended attributes, POSIX ACLs, and file capabilities are collected from the filesystem via the FileSystemAccessor interface and encoded into archives. Metadata change detection compares all extended metadata fields (stat, xattrs, ACLs, fcaps) to trigger re-upload when they change.

PBS Reader Protocol

For restoring backups, the PBSReader type provides access to the Proxmox Backup Server reader protocol (proxmox-backup-reader-protocol-v1) via HTTP/2. This enables efficient downloading of:

  • Index files (.didx, .fidx, .blob) via GET /download
  • Individual chunks by digest via GET /chunk

The reader integrates with the datastore.Restorer to reconstruct files from their chunks, supporting both full file restoration and partial/range reads.

Example: 

reader := backupproxy.NewPBSReader(cfg, "host", "mybackup", backupTime)
if err := reader.Connect(ctx); err != nil {
    return err
}
defer reader.Close()

// Download index
didxData, _ := reader.DownloadFile("root.pxar.didx")
idx, _ := datastore.ReadDynamicIndex(didxData)

// Restore entire file
var buf bytes.Buffer
reader.RestoreFile(idx, &buf)

// Or restore just a range (e.g., bytes 1024-2048)
reader.RestoreFileRange(idx, 1024, 1024, &buf)

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EntryMatches added in v0.5.0 

func EntryMatches(current DirEntry, catalogMetadata pxar.Metadata, prev *CatalogEntry) bool

EntryMatches checks if a current directory entry matches a catalog entry for metadata change detection. Returns true if the file metadata hasn't changed. Compares stat fields, file type, size, xattrs, ACLs, FCaps, and QuotaProjectID.

Types

type BackupConfig 

type BackupConfig struct {
	BackupType  datastore.BackupType // vm, ct, or host
	BackupID    string               // backup identifier
	BackupTime  int64                // Unix timestamp for this snapshot
	Namespace   string               // optional namespace
	Compress    bool                 // compress chunks with zstd
	ChunkConfig buzhash.Config       // buzhash chunking parameters

	// DetectionMode controls the backup format and change detection strategy.
	// DetectionLegacy (default) creates a single v1 archive with all data.
	// DetectionData creates split v2 archives with all data re-encoded.
	// DetectionMetadata creates split v2 archives, reusing payload chunks
	// for files whose metadata hasn't changed since PreviousBackup.
	DetectionMode DetectionMode

	// PreviousBackup identifies the snapshot to compare against when
	// DetectionMode is DetectionMetadata. Required for metadata mode.
	PreviousBackup *PreviousBackupRef

	// CryptMode controls encryption of backup data.
	// CryptModeNone (default) stores data in cleartext.
	// CryptModeEncrypt encrypts all data with AEAD.
	// CryptModeSign signs the manifest without encrypting data.
	CryptMode datastore.CryptMode

	// CryptConfig provides the encryption keys for CryptModeEncrypt or CryptModeSign.
	// Must be set when CryptMode is not CryptModeNone.
	CryptConfig *datastore.CryptConfig
}

BackupConfig holds parameters for a single backup operation.

type BackupResult 

type BackupResult struct {
	Manifest        *datastore.Manifest
	TotalBytes      int64
	FileCount       int
	DirCount        int
	Duration        time.Duration
	CatalogUploaded bool
}

BackupResult describes the outcome of a backup operation.

type BackupSession 

type BackupSession interface {
	UploadArchive(ctx context.Context, name string, data io.Reader) (*UploadResult, error)
	UploadSplitArchive(ctx context.Context, metadataName string, metadataData io.Reader, payloadName string, payloadData io.Reader) (*SplitArchiveResult, error)
	UploadBlob(ctx context.Context, name string, data []byte) error
	Finish(ctx context.Context) (*datastore.Manifest, error)
}

BackupSession represents an active backup upload session.

type Catalog added in v0.5.0 

type Catalog map[string]*CatalogEntry

Catalog is a map from normalized file path to CatalogEntry.

func BuildCatalog added in v0.5.0 

func BuildCatalog(metaIdx *datastore.DynamicIndexReader, chunkSource datastore.ChunkSource) (Catalog, error)

BuildCatalog constructs a metadata catalog from a previous backup's .mpxar.didx and .ppxar.didx indexes. It restores the metadata stream from chunks and walks it with the accessor to extract file entries.

type CatalogEntry added in v0.5.0 

type CatalogEntry struct {
	Path          string
	Stat          format.Stat
	Metadata      pxar.Metadata
	FileSize      uint64
	PayloadOffset uint64
	IsRegularFile bool
}

CatalogEntry holds metadata from a previous backup's .mpxar archive, used for metadata change detection.

type ClientProvider 

type ClientProvider interface {
	Stat(ctx context.Context, path string) (format.Stat, error)
	ReadDir(ctx context.Context, path string) ([]DirEntry, error)
	ReadFile(ctx context.Context, path string, offset, length int64) ([]byte, error)
	ReadLink(ctx context.Context, path string) (string, error)

	// Extended metadata methods for full archive fidelity.
	GetXAttrs(ctx context.Context, path string) ([]format.XAttr, error)
	GetACL(ctx context.Context, path string) (pxar.ACL, error)
	GetFCaps(ctx context.Context, path string) ([]byte, error)
}

ClientProvider is the interface the server uses to access client data. Transport implementations bridge this to the actual network. Each method call corresponds to one request-response round trip.

type DetectionMode added in v0.5.0 

type DetectionMode int

DetectionMode controls how file changes are detected between backup runs. 

const (
	// DetectionLegacy creates a single self-contained pxar v1 archive.
	// All file data and metadata are read and encoded in one stream.
	DetectionLegacy DetectionMode = iota

	// DetectionData creates split pxar v2 archives (.mpxar + .ppxar).
	// All file data is still read fully, but metadata and payload are
	// stored in separate streams for more efficient catalog access.
	DetectionData

	// DetectionMetadata creates split pxar v2 archives (.mpxar + .ppxar)
	// but only re-reads file content for files whose metadata (size, mtime,
	// uid, gid, mode, xattrs) has changed since the previous backup.
	// Unchanged files reuse payload chunks from the previous backup.
	DetectionMetadata
)

func (DetectionMode) String added in v0.5.0 

func (d DetectionMode) String() string

type DirEntry 

type DirEntry struct {
	Name string
	Stat format.Stat
	Size uint64 // file size in bytes (0 for non-regular files)

	// Extended metadata for accurate change detection and full archive fidelity.
	// Populated from FileSystemAccessor when available.
	XAttrs         []format.XAttr
	ACL            pxar.ACL
	FCaps          []byte
	QuotaProjectID *uint64
}

DirEntry represents a single entry from a directory listing on the client.

type FileSystemAccessor 

type FileSystemAccessor interface {
	Stat(path string) (format.Stat, error)
	ReadDir(path string) ([]DirEntry, error)
	ReadFile(path string, offset, length int64) ([]byte, error)
	ReadLink(path string) (string, error)

	// GetXAttrs returns the extended attributes for the given path.
	GetXAttrs(path string) ([]format.XAttr, error)

	// GetACL returns the POSIX ACL entries for the given path.
	GetACL(path string) (pxar.ACL, error)

	// GetFCaps returns the file capabilities for the given path.
	GetFCaps(path string) ([]byte, error)
}

FileSystemAccessor is the interface the client uses to access the local filesystem. Users provide their own implementation. This indirection allows testing without a real filesystem and running in constrained environments.

type LocalClient 

type LocalClient struct {
	// contains filtered or unexported fields
}

LocalClient implements ClientProvider by delegating to a FileSystemAccessor. This is the client-side component: runs on the machine being backed up.

func NewLocalClient 

func NewLocalClient(fs FileSystemAccessor) *LocalClient

NewLocalClient creates a client backed by the given FileSystemAccessor.

func (*LocalClient) GetACL added in v0.6.0 

func (lc *LocalClient) GetACL(_ context.Context, path string) (pxar.ACL, error)

GetACL returns the POSIX ACL entries for the given path.

func (*LocalClient) GetFCaps added in v0.6.0 

func (lc *LocalClient) GetFCaps(_ context.Context, path string) ([]byte, error)

GetFCaps returns the file capabilities for the given path.

func (*LocalClient) GetXAttrs added in v0.6.0 

func (lc *LocalClient) GetXAttrs(_ context.Context, path string) ([]format.XAttr, error)

GetXAttrs returns the extended attributes for the given path.

func (*LocalClient) ReadDir 

func (lc *LocalClient) ReadDir(_ context.Context, path string) ([]DirEntry, error)

ReadDir returns directory entries for the given path by delegating to the underlying FileSystemAccessor.

func (*LocalClient) ReadFile 

func (lc *LocalClient) ReadFile(_ context.Context, path string, offset, length int64) ([]byte, error)

ReadFile returns file content at the given path and offset by delegating to the underlying FileSystemAccessor. 

func (lc *LocalClient) ReadLink(_ context.Context, path string) (string, error)

ReadLink returns the symlink target for the given path by delegating to the underlying FileSystemAccessor.

func (*LocalClient) Stat 

func (lc *LocalClient) Stat(_ context.Context, path string) (format.Stat, error)

Stat returns file metadata for the given path by delegating to the underlying FileSystemAccessor.

type LocalStore 

type LocalStore struct {
	// contains filtered or unexported fields
}

LocalStore implements RemoteStore using a local filesystem directory. It uses datastore.ChunkStore for chunk storage and writes index/blob files to disk. Intended for testing and offline backups.

func NewLocalStore 

func NewLocalStore(baseDir string, config buzhash.Config, compress bool) (*LocalStore, error)

NewLocalStore creates a LocalStore backed by the given directory.

func (*LocalStore) NewPreviousSnapshotSource added in v0.5.0 

func (ls *LocalStore) NewPreviousSnapshotSource(_ context.Context, _ datastore.BackupType, _ string, _ int64, _ string) (PreviousSnapshotSource, error)

NewPreviousSnapshotSource creates a PreviousSnapshotSource for a local backup snapshot.

func (*LocalStore) ReadPreviousArchive added in v0.5.0 

func (ls *LocalStore) ReadPreviousArchive(_ context.Context, _ datastore.BackupType, _ string, _ int64, _, filename string) ([]byte, error)

ReadPreviousArchive reads an archive file from a previous local backup snapshot.

func (*LocalStore) StartSession 

func (ls *LocalStore) StartSession(_ context.Context, config BackupConfig) (BackupSession, error)

StartSession creates a new local backup session.

type NoExtendedAttrs added in v0.6.0 

type NoExtendedAttrs struct{}

NoExtendedAttrs provides no-op implementations of the extended metadata methods. Embed this in FileSystemAccessor implementations that don't support xattrs, ACLs, or file capabilities.

func (NoExtendedAttrs) GetACL added in v0.6.0 

func (NoExtendedAttrs) GetACL(string) (pxar.ACL, error)

func (NoExtendedAttrs) GetFCaps added in v0.6.0 

func (NoExtendedAttrs) GetFCaps(string) ([]byte, error)

func (NoExtendedAttrs) GetXAttrs added in v0.6.0 

func (NoExtendedAttrs) GetXAttrs(string) ([]format.XAttr, error)

type PBSConfig 

type PBSConfig struct {
	BaseURL       string // PBS API base URL (e.g. "https://pbs:8007/api2/json")
	Datastore     string // target datastore name
	AuthToken     string // PBS API token ("TOKENID:SECRET")
	SkipTLSVerify bool   // disable TLS certificate verification
	Namespace     string // optional namespace for the backup
}

PBSConfig holds configuration for connecting to a Proxmox Backup Server.

type PBSReader 

type PBSReader struct {
	// contains filtered or unexported fields
}

PBSReader provides read access to a PBS datastore via the reader protocol.

func NewPBSReader 

func NewPBSReader(config PBSConfig, backupType, backupID string, backupTime int64) *PBSReader

NewPBSReader creates a new PBS reader for the given backup snapshot.

func (*PBSReader) AsChunkSource 

func (r *PBSReader) AsChunkSource() datastore.ChunkSource

AsChunkSource returns a ChunkSource interface for the restorer.

func (*PBSReader) Close 

func (r *PBSReader) Close() error

Close closes the reader connection.

func (*PBSReader) Connect 

func (r *PBSReader) Connect(ctx context.Context) error

Connect establishes the H2 reader connection to PBS.

func (*PBSReader) DownloadChunk 

func (r *PBSReader) DownloadChunk(digest [32]byte) ([]byte, error)

DownloadChunk downloads a chunk by its digest. The reader protocol requires that the index file referencing this chunk has been downloaded first (via DownloadFile), which populates the server-side allowed_chunks set.

func (*PBSReader) DownloadFile 

func (r *PBSReader) DownloadFile(fileName string) ([]byte, error)

DownloadFile downloads an index file (.didx, .fidx, .blob) from PBS.

func (*PBSReader) RestoreFile 

func (r *PBSReader) RestoreFile(idx *datastore.DynamicIndexReader, w io.Writer) error

RestoreFile restores a complete file from a dynamic index. This downloads all chunks and reconstructs the file content. The index file must have been downloaded first (via DownloadFile) to populate the server-side allowed_chunks set.

func (*PBSReader) RestoreFileRange 

func (r *PBSReader) RestoreFileRange(idx *datastore.DynamicIndexReader, offset, length uint64, w io.Writer) error

RestoreFileRange restores a specific byte range from a file. Useful for partial reads without downloading the entire file.

type PBSRemoteStore 

type PBSRemoteStore struct {
	// contains filtered or unexported fields
}

PBSRemoteStore implements RemoteStore via the PBS H2 backup protocol.

func NewPBSRemoteStore 

func NewPBSRemoteStore(config PBSConfig, chunkCfg buzhash.Config, compress bool) *PBSRemoteStore

NewPBSRemoteStore creates a PBS remote store with the given configuration.

func (*PBSRemoteStore) NewPreviousSnapshotSource added in v0.5.0 

func (ps *PBSRemoteStore) NewPreviousSnapshotSource(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace string) (PreviousSnapshotSource, error)

NewPreviousSnapshotSource creates a PreviousSnapshotSource connected to a PBS snapshot.

func (*PBSRemoteStore) ReadPreviousArchive added in v0.5.0 

func (ps *PBSRemoteStore) ReadPreviousArchive(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace, filename string) ([]byte, error)

ReadPreviousArchive reads an archive file from a previous PBS backup snapshot.

func (*PBSRemoteStore) StartSession 

func (ps *PBSRemoteStore) StartSession(ctx context.Context, config BackupConfig) (BackupSession, error)

StartSession dials PBS via H2 upgrade and returns a backup session.

type PreviousBackupRef added in v0.5.0 

type PreviousBackupRef struct {
	BackupType datastore.BackupType
	BackupID   string
	BackupTime int64
	Namespace  string
	Dir        string // local directory containing previous snapshot files (for LocalStore)
}

PreviousBackupRef identifies a previous backup snapshot for metadata comparison.

type PreviousSnapshotSource added in v0.5.0 

type PreviousSnapshotSource interface {
	ReadArchive(filename string) ([]byte, error)
	ChunkSource() datastore.ChunkSource
	Close() error
}

PreviousSnapshotSource provides read access to a previous backup snapshot for metadata change detection. It can read archive files and download chunks.

func NewPreviousSnapshotSourceFromDir added in v0.5.0 

func NewPreviousSnapshotSourceFromDir(dir string) (PreviousSnapshotSource, error)

NewPreviousSnapshotSourceFromDir creates a PreviousSnapshotSource from a local directory.

type RemoteStore 

type RemoteStore interface {
	RemoteStoreBase
	SnapshotReader
}

RemoteStore abstracts the backup storage backend.

type RemoteStoreBase added in v0.5.0 

type RemoteStoreBase interface {
	StartSession(ctx context.Context, config BackupConfig) (BackupSession, error)
}

RemoteStoreBase contains the session creation method.

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server orchestrates pull backups: walks the client filesystem, encodes a pxar archive, chunks it with buzhash, and uploads to a RemoteStore.

func NewServer 

func NewServer(client ClientProvider, store RemoteStore) *Server

NewServer creates a backup server with the given client provider and store.

func (*Server) RunBackup 

func (s *Server) RunBackup(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunBackup executes a full pull backup using the legacy v1 format (single archive).

func (*Server) RunBackupWithMode added in v0.5.0 

func (s *Server) RunBackupWithMode(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunBackupWithMode dispatches to the appropriate backup method based on config.DetectionMode. It uses RunBackup for legacy, RunSplitBackup for data, and RunMetadataBackup for metadata mode.

func (*Server) RunMetadataBackup added in v0.5.0 

func (s *Server) RunMetadataBackup(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunMetadataBackup executes an incremental pull backup using metadata change detection. It downloads the previous backup's metadata and payload catalogs, compares current file metadata against them, and only reads content from the client for files that changed. Unchanged files reuse their payload data from the previous backup.

func (*Server) RunSplitBackup added in v0.4.0 

func (s *Server) RunSplitBackup(ctx context.Context, root string, config BackupConfig) (*BackupResult, error)

RunSplitBackup executes a full pull backup using the split archive format (v2, data mode). The encoder writes metadata and payload to buffers first, then uploads them sequentially via UploadSplitArchive. This avoids the io.Pipe deadlock that occurs when UploadSplitArchive reads one stream at a time while the encoder writes both simultaneously.

type SnapshotReader added in v0.5.0 

type SnapshotReader interface {
	ReadPreviousArchive(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace, filename string) ([]byte, error)
	NewPreviousSnapshotSource(ctx context.Context, backupType datastore.BackupType, backupID string, backupTime int64, namespace string) (PreviousSnapshotSource, error)
}

SnapshotReader can read files from previous snapshots.

type SplitArchiveResult added in v0.4.0 

type SplitArchiveResult struct {
	MetadataResult *UploadResult
	PayloadResult  *UploadResult
}

SplitArchiveResult contains the results of uploading a split archive. The metadata and payload are uploaded as separate .didx files.

type UploadResult 

type UploadResult struct {
	Filename string   // e.g., "root.pxar.didx"
	Size     uint64   // total index size
	Digest   [32]byte // SHA-256 of the index
}

UploadResult describes the outcome of an archive upload.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.