gopbs

README

gopbs

GoPBS is a pure-Go library for backing up data to Proxmox Backup Server (PBS). It generates PXAR archives (v1 and the split v2 format) and .pcat1 catalogs, chunks and deduplicates content exactly like the official client, and uploads over the PBS HTTP/2 backup protocol — either as one gopbs.Backup call or through the individual building blocks (archive, pxar, catalog, chunker, scan, pbs).

Output is validated byte-for-byte against the official pxar tool, and snapshots restore with the stock proxmox-backup-client. Full Linux metadata fidelity: ownership, modes, timestamps (ns), xattrs, POSIX ACLs, file capabilities, quota project IDs, hardlinks, symlinks, device nodes, FIFOs and sockets.

Credit

Credit where credit is due - I did not figure most of this process out by myself.

A large part of the initial work regarding finding the specification, optimizing the backup process, etc. was done by @tizbac, go support his work if you are able.

Performance

We include performance tests (cd tests && GOPBS_PERF=1 GOPBS_PERF_MB=512 go test -tags integration -run TestPerformance -v .). The results are roughly as follows:

Tooling	Format	Mode	Time	Throughput
gopbs	v1	sync (workers=1)	2.81s	182.0 MiB/s
gopbs	v1	async (workers=8)	0.79s	649.9 MiB/s
gopbs	v2	async	0.76s	676.9 MiB/s
official pxar cli	v1	docker	7.28s	70.3 MiB/s
official pxar cli	v2	docker	4.22s	121.5 MiB/s
gopbs	v1	upload (fresh)	3.72s	137.6 MiB/s
gopbs	v1	upload (dedup rerun)	1.93s	265.6 MiB/s
gopbs	v2	upload (fresh)	2.72s	188.4 MiB/s
gopbs	v2	upload (dedup rerun)	1.82s	281.5 MiB/s
official client	v1	upload (fresh)	3.03s	168.9 MiB/s
official client	v1	upload (dedup rerun)	1.78s	288.1 MiB/s
official client	v2	upload (fresh)	3.58s	142.8 MiB/s
official client	v2	upload (dedup rerun)	0.79s	651.9 MiB/s

Installation

go get github.com/osshield/gopbs

A small CLI for generating archives to files (used by the integration harness) ships as well:

go install github.com/osshield/gopbs/cmd/gopbs-pxar@latest
gopbs-pxar -catalog out.pcat1 out.pxar /some/dir                 # v1 + catalog
gopbs-pxar -payload out.ppxar out.mpxar /some/dir                # v2 split archive

Quickstart: full backup

One call — scan, plan, generate, chunk, deduplicate against the previous snapshot, upload, catalog, manifest, finish. Everything is streamed; nothing is written to disk.

result, err := gopbs.Backup(context.Background(), gopbs.BackupOptions{
    Client: pbs.Config{
        BaseURL:     "https://pbs.example.com:8007",
        Auth:        pbs.TokenAuth{AuthID: "backup@pbs!mytoken", Secret: "…"},
        Fingerprint: "AA:BB:…", // sha256 fingerprint of the server certificate
        Datastore:   "backups",
    },
    Archive: archive.Options{
        Name: "root",
        Scan: scan.Options{
            SkipOnError: true,
            Exclude:     []string{"*.tmp", "!keep.tmp", "cache/"}, // proxmox-backup-client --exclude syntax
        },
    },
    Ref:   pbs.SnapshotRef{Type: "host", ID: "myhost"},
    Paths: []string{"/etc", "/srv"},
})

PasswordAuth (ticket login) works in place of TokenAuth. A single directory in Paths becomes the archive root; multiple paths (and Streams — virtual files backed by io.Readers) are placed under a virtual root named after the archive. Set OnProgress for live per-chunk progress, and read non-fatal events (skipped entries, files that changed mid-read) from result.Warnings.

Excluding entries. Scan.Exclude takes patterns in the official client's --exclude syntax: a leading / anchors to the archive root, a trailing / matches directories only, ! re-includes, ** spans directory levels, and the last matching pattern wins. They are recorded in the archive exactly as proxmox-backup-client does (a .pxarexclude-cli file in the v1 root, the prelude record in v2). Set Scan.PxarExcludeFiles to honour .pxarexclude files found in the tree, and Scan.Filter for arbitrary logic (size, age, device boundaries) that leaves no trace in the archive. See docs/api.md.

Set Format: gopbs.FormatV2 for split archives: metadata (.mpxar) and payload (.ppxar) upload as two concurrent indexes, so metadata-only changes (touched mtimes, permissions) leave the payload stream — by far the larger one — fully deduplicated. Restore requires proxmox-backup-client ≥ 3.2; v1 is the safe default for older servers and pmoxs3backuproxy.

Quickstart: archive only

The archive package generates PXAR streams without any server involved:

a, _ := archive.New(archive.Options{})
_ = a.AddDirectory("/etc")

stream, _ := a.GenerateV1(context.Background())
defer stream.Close()

out, _ := os.Create("/tmp/etc.pxar")
defer out.Close()
io.Copy(out, stream)

GenerateCatalog produces the matching .pcat1, GenerateV2 the split stream pair (consume both concurrently), and EstimatedSizeV1/EstimatedSizeV2 exact size estimates from scan metadata. Generation is asynchronous by default — a worker pool reads file contents in parallel and a reorder buffer keeps the output byte-identical to synchronous mode (Workers: 1). Payload sizes are bound late (open+fstat at dispatch), so files that change size between scan and read never corrupt the archive: content is padded/truncated to the committed size and reported as a warning.

Runnable programs for all of this live in examples/: sync, async, multi-stream and v2 backups, each with a live progress bar.

Feature Status

• PXAR Archive generation

  • PXAR Version 1
  • PXAR Version 2 (split .mpxar/.ppxar, byte-identical to pxar create --payload-output)
  • PCAT generation
  • Synchronous and asynchronous archive generation
  • Compression (zstd, per chunk — as PBS stores data)
  • Encryption (AES-256-GCM, key-file compatible with proxmox-backup-client, signed manifests, sign-only mode, RSA master-key wrapping)
  • Exclusions (--exclude pattern syntax of proxmox-backup-client, .pxarexclude files, custom filter callback; recorded in the archive like the official client, byte-identical output)

• Backup to PBS and @tizbac/pmoxs3backuproxy (S3-compatible proxy for PBS)

  • Directories (single or multiple)
  • Files (single file or multiple files per archive)
  • Data stream (stdin, readers)*
  • Synchronous and parallel backup uploads
  • Deduplication and change-only uploads

• Streams are implemented, but if the size of the stream changes (increases) after the backup operation has started for that stream, there will be a loss of data. Always make sure the stream doesn't get any more data after backups have begun.

• Restore data from PBS

  • Restore to disk
  • Restore to stream (stdout, writers)
  • Synchronous and parallel restore

• Backup and restore sources

  • Linux filesystem

    • Plain Filesystem (ext4, xfs, btrfs, etc.)
    • Snapshot-based (LVM, ZFS, BTRFS, etc.)
    • Remote filesystems (via rclone)
    • Block devices

  • Windows filesystem

    • Plain Filesystem (NTFS, FAT32, exFAT, etc.)
    • Snapshot-based (VSS)

  • Backup and restore databases

    • MySQL
    • PostgreSQL
    • MongoDB
    • SQL Server

  • Backup and restore kubernetes

    • Backup PVCs (non-snapshot)
    • Backup snapshots (via CSI and Snapshot API)
    • Backup cluster resources/state

Documentation

• docs/api.md — API guide: how the packages fit together, the one-call path, archive-only use, the client on its own, concurrency notes.
• docs/pxar-format.md — the PXAR v1/v2 formats and the .pcat1 catalog, byte for byte.
• docs/wire-protocol.md — the PBS backup wire protocol: auth, upgrade handshake, endpoints, blob framing, didx, manifest.
• docs/integration-harness.md — the docker test stack, what each test proves, the performance comparison.
• ARCHITECTURE.md — design rationale; IMPLEMENTATION.md — phase-by-phase build-out with verification notes.

The integration harness in tests/ byte-compares gopbs output against the official pxar CLI and tizbac's client, restores snapshots with the official proxmox-backup-client from a pmoxs3backuproxy+garage stack, and verifies full end-to-end backups including second-run deduplication:

cd tests && go test -tags integration .

Fully annotated example (all options)

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"time"

	"github.com/osshield/gopbs"
	"github.com/osshield/gopbs/archive"
	"github.com/osshield/gopbs/pbs"
	"github.com/osshield/gopbs/scan"
)

func main() {
	started := time.Now()

	result, err := gopbs.Backup(context.Background(), gopbs.BackupOptions{
		Client: pbs.Config{
			// The URL of the PBS instance, including https and port
			BaseURL: "https://my-pbs.instance.localhost:8007",

			// Authenticate with AuthID and Secret (token-based)
			Auth: pbs.TokenAuth{
				AuthID: "root@pbs!token",
				Secret: "12345678-1234-1234-1234-1234567890ab",
			},

			// You can also authenticate with username/password
			// Auth: pbs.PasswordAuth{
			//     Username: "root",
			//     Realm: "pbs",
			//     Password: "my-secure-password"
			// },

			// The datastore to use for the backup
			Datastore: "SRV-TESTS",

			// Optionally, you can specify a different namespace
			// Namespace:       "my-namespace",

			// The fingerprint of the PBS server's TLS certificate (SHA256)
			Fingerprint: "2F:6B:30:B8:B4:8B:5C:95:40:B9:F0:06:4B:5F:62:6A:47:E9:8B:27:EE:09:96:0C:30:F1:22:AE:58:B9:A5:24",

			// Alternatively, you can skip TLS verification (not recommended for production)
			// InsecureSkipAll: true,

			// Number of concurrent workers for uploading chunks to PBS.
			// Default: 4
			Workers: 6,

			// Set the average chunk size, must be a power of two
			// Generally, this does not need to be changed, but can be adjusted for performance tuning
			// in some specific scenarios. The default value is 4MiB, which balances speed and efficiency for deduplication.
			// Default: 4MiB
			// ChunkSizeAvg: 4 * 1024 * 1024,

			// If you want to follow the upload progress, you can use the OnProgress callback.
			// OnUploadProgress: func(archiveName string, stats pbs.UploadStats, done bool) {}
		},

		Archive: archive.Options{
			// The archive ID (name)
			Name: "my-archive-name",

			// Number of concurrent workers for reading file contents.
			// 0 = GOMAXPROCS, 1 = fully synchronous generation.
			Workers: 5,

			// Settings for scanning and creating the archive
			Scan: scan.Options{
				// Skip files on error instead of failing the entire backup
				SkipOnError: true,

				// Replaces the metadata reader with a custom implementation instead of the linux stat reader.
				// This is useful for testing environments or to add metadata to a stream-based backup
				// Reader: scan.MetadataReader(nil),

				// If true, skip reading quota project IDs from the filesystem
				// This can improve performance, and saves one ioctl call per directory and file,
				// but it means that the backup will not be able to restore quota project IDs.
				// SkipQuotaProjIDs: false,

				// Exclude patterns in proxmox-backup-client --exclude syntax, matched against
				// archive-relative paths; last match wins. Recorded in the archive like the
				// official client does (.pxarexclude-cli in v1, the prelude in v2).
				Exclude: []string{"*.tmp", "!keep.tmp", "cache/", "/var/cache", "**/node_modules"},

				// Honour .pxarexclude files found in the scanned directories (off by default).
				// PxarExcludeFiles: true,

				// Arbitrary exclusion logic; return false to omit the entry and its subtree.
				// Not recorded in the archive.
				// Filter: func(path, archivePath string, st scan.Stat) bool { return st.Size < 1<<30 },

				// Called for every entry left out by Exclude, .pxarexclude or Filter.
				// OnExclude: func(path, archivePath string) { log.Println("excluded", path) },
			},

			// This function is called for each warning encountered during archive generation.
			// Warnings are also collected into result.Warnings.
			OnWarn: func(w archive.Warning) {
				fmt.Fprintf(os.Stderr, "warning: %s (kind %d, err %v)\n", w.Path, w.Kind, w.Err)
			},

			// The memory budget for the async reorder buffer.
			// Default: 64MiB
			// Buffer: 64 * 1024 * 1024,
		},
		Ref: pbs.SnapshotRef{
			// Set the snapshot/backup type. Valid values are "host", "vm", or "ct". Default is "host".
			Type: "ct",

			// Set the ID for the snapshot. If empty, it defaults to the hostname of the machine running the backup.
			ID: "my-container-name",

			// Manually set the timestamp for the backup snapshot.
			// Default: time.Now()
			// Time: time.Now(),
		},

		// Set the archive format for uploads.
		// FormatV1 is the classic single-stream pxar plus a .pcat1 catalog, supported everywhere.
		// FormatV2 is the split .mpxar/.ppxar format: better deduplication for metadata-only
		// changes, requires proxmox-backup-client >= 3.2 to restore.
		Format: gopbs.FormatV1,

		// Filesystem paths to include in the backup.
		// If more than one is selected, a virtual top-level directory will be created with the name of the archive.
		// If a single directory is selected, it will become the root of the archive.
		Paths: []string{"/srv", "/var"},

		// Virtual files backed by readers, placed alongside Paths under the virtual root.
		// Streams: []gopbs.Stream{{Name: "dump.sql", Size: size, Reader: r}},

		// Metadata blobs stored in the snapshot alongside the archive (not part of it) —
		// for backup metadata PBS has no native place for. Restored with
		// `proxmox-backup-client restore <snapshot> app-meta.json.blob <target>`.
		// Blobs: []gopbs.Blob{{Name: "app-meta.json", Data: metaJSON}},

		// Callback for live progress updates during the backup process.
		// OnProgress: func(p gopbs.Progress) {},

		// Skip the PCAT catalogue upload. This means the snapshot will not be browsable in the PBS UI.
		// FormatV1 only; v2 has no catalog.
		// SkipCatalog: false,
	})
	if err != nil {
		log.Fatalf("%v", err)
	}

	fmt.Printf("backed up %s/%s/%s: %d bytes in %d chunks (%d uploaded, %d deduplicated), catalog %d bytes, %d warnings (%.1fs)\n",
		result.Ref.Type, result.Ref.ID, result.Ref.Time.UTC().Format(time.RFC3339),
		result.Archive.Size, result.Archive.ChunkCount,
		result.Archive.NewChunks, result.Archive.ReusedChunks,
		result.Catalog.Size, len(result.Warnings), time.Since(started).Seconds())
}

License

See LICENSE.

Documentation

Overview

Package gopbs creates PXAR archives (Proxmox archive format, versions 1 and 2) and uploads them to Proxmox Backup Server.

The subpackages are usable independently:

• pxar: the pure PXAR format layer (record encoders, goodbye tables)
• catalog: the .pcat1 catalog encoder (v1 archives)
• scan: filesystem scanning with full metadata recording (Linux only currently)
• archive: archive planning and (async) stream generation
• chunker: the buzhash chunker used by PBS
• pbs: the Proxmox Backup Server client (backup sessions, chunk upload)

This package ties them together with Backup: one call that scans, plans, generates, chunks, deduplicates and uploads with no intermediate files.

Index

• type BackupOptions
• type BackupResult
  • func Backup(ctx context.Context, opts BackupOptions) (*BackupResult, error)
• type Blob
• type Format
• type Progress
• type Stream

Examples

• Backup

Types

type BackupOptions

type BackupOptions struct {
	// Client configures the PBS connection (server, auth, datastore,
	// namespace, upload workers, chunk size). Set Client.Crypt to encrypt
	// the backup client-side (or sign its manifest); see pbs.CryptConfig.
	Client pbs.Config
	// Archive configures generation (name, workers, buffer, scan policy).
	// Exclusions live in Archive.Scan: Exclude patterns in
	// proxmox-backup-client syntax (recorded in the archive like the
	// official client's --exclude: a .pxarexclude-cli root file in v1, the
	// prelude in v2), PxarExcludeFiles to honour .pxarexclude files in the
	// tree, and Filter for arbitrary caller logic that leaves no trace in
	// the archive. Invalid patterns fail before any connection is made.
	// Its OnWarn is honored in addition to warnings being collected into
	// the result.
	Archive archive.Options
	// Ref identifies the snapshot; zero-value fields get the usual defaults
	// (type "host", hostname id, current time).
	Ref pbs.SnapshotRef
	// Format selects the archive format (FormatV1 or FormatV2).
	Format Format
	// Paths are the directories and files to back up. A single directory
	// becomes the archive root; any other combination lives under a virtual
	// root named Archive.Name.
	Paths []string
	// Streams are virtual files backed by readers, placed under the virtual
	// root alongside Paths. Readers are consumed during the backup, so a
	// BackupOptions value with streams is good for one call.
	Streams []Stream
	// Blobs are uploaded as separate blob files in the snapshot, alongside
	// the archive indexes and listed in the manifest — they are not part of
	// the archive. Use them for backup metadata PBS has no native place for
	// (application state, retention hints, tool versions, …). Contents are
	// downloaded back with `proxmox-backup-client restore <snapshot>
	// <name>.blob <target>`.
	Blobs []Blob
	// SkipCatalog omits the .pcat1 upload (the snapshot will not be
	// browsable in the PBS UI). FormatV1 only; v2 has no catalog.
	SkipCatalog bool
	// OnProgress, when set, receives live upload progress: one call per
	// committed chunk plus a final call with Done=true per index. Setting it
	// costs one extra metadata scan (the size estimate for Total). Called
	// from the uploading goroutine; keep it fast. FormatV2 uploads its two
	// indexes concurrently, so calls can arrive from two goroutines at once —
	// synchronize any shared state.
	OnProgress func(Progress)
}

BackupOptions configures one Backup call.

type BackupResult

type BackupResult struct {
	// Ref is the snapshot identity as written (defaults resolved) — what a
	// restore addresses.
	Ref pbs.SnapshotRef
	// ArchiveName is the uploaded index name: "<base>.pxar.didx" for v1, the
	// metadata stream's "<base>.mpxar.didx" for v2.
	ArchiveName string
	// Archive is the pxar index's upload stats (the metadata stream's, for
	// v2).
	Archive pbs.UploadStats
	// Catalog is the .pcat1 index's upload stats (v1 only).
	Catalog pbs.UploadStats
	// Payload is the payload stream's ("<base>.ppxar.didx") upload stats
	// (v2 only).
	Payload pbs.UploadStats
	// Warnings collects the non-fatal generation events (skipped entries,
	// padded/truncated files, torn reads).
	Warnings []archive.Warning
}

BackupResult reports a finished backup.

func Backup

func Backup(ctx context.Context, opts BackupOptions) (*BackupResult, error)

Backup performs one complete backup: scan → plan → generate → chunk → deduplicate → upload → catalog → manifest → finish, fully streamed.

Example

A complete backup in one call: scan, plan, generate, chunk, deduplicate against the previous snapshot, upload, catalog, manifest, finish.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/osshield/gopbs"
	"github.com/osshield/gopbs/archive"
	"github.com/osshield/gopbs/pbs"
	"github.com/osshield/gopbs/scan"
)

func main() {
	result, err := gopbs.Backup(context.Background(), gopbs.BackupOptions{
		Client: pbs.Config{
			BaseURL:     "https://pbs.example.com:8007",
			Auth:        pbs.TokenAuth{AuthID: "backup@pbs!mytoken", Secret: "…"},
			Fingerprint: "AA:BB:…", // sha256 of the server certificate
			Datastore:   "backups",
		},
		Archive: archive.Options{
			Name: "root",
			Scan: scan.Options{SkipOnError: true},
		},
		Ref:   pbs.SnapshotRef{Type: "host", ID: "myhost"},
		Paths: []string{"/etc", "/srv"},
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s/%s/%s: %d bytes in %d chunks (%d new)\n",
		result.Ref.Type, result.Ref.ID, result.Ref.Time,
		result.Archive.Size, result.Archive.ChunkCount, result.Archive.NewChunks)
}

type Blob

type Blob struct {
	Name string
	Data []byte
}

Blob is a metadata file for BackupOptions.Blobs: stored in the snapshot as "<Name>.blob" (the suffix is appended when missing), zstd-compressed when that is smaller. Data is held in memory — blobs are for small metadata; bulk content belongs in the archive or a Stream. The name "index.json" is reserved for the manifest, "rsa-encrypted.key" for the wrapped encryption key. Blobs are encrypted when Client.Crypt is in encrypt mode.

type Format

type Format int

Format selects the pxar archive format.

const (
	// FormatV1 is the classic single-stream pxar plus a .pcat1 catalog.
	FormatV1 Format = iota
	// FormatV2 is the split format: a metadata stream (.mpxar) and a payload
	// stream (.ppxar) uploaded as two dynamic indexes. No catalog — the
	// metadata stream itself serves browsing. Requires proxmox-backup-client
	// >= 3.2 to restore.
	FormatV2
)

type Progress

type Progress struct {
	// Archive is the index being uploaded (e.g. "root.pxar.didx" or
	// "catalog.pcat1.didx").
	Archive string
	// Total is the estimated byte size of this index from scan metadata
	// (0 = unknown; the catalog has no estimate). The actual size may
	// differ slightly when files change during the backup.
	Total uint64
	// Stats carries the live counters; Stats.Size is the number of bytes
	// committed so far and grows monotonically per index.
	Stats pbs.UploadStats
	// Done marks the final report for this index.
	Done bool
}

Progress is one live progress report during Backup.

type Stream

type Stream struct {
	Name   string
	Size   int64
	Reader io.Reader
}

Stream is a virtual file for BackupOptions.Streams. Size must be declared up front (the archive layout commits to it); a reader yielding a different byte count is padded or truncated to Size with a warning.