restore

package
v0.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 6, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MaxPrefetchBytes caps the estimated frame bytes held by one window.
	MaxPrefetchBytes = 16 << 20
	// MaxPrefetchChunks caps the chunks in one window, so pathologically small
	// chunks cannot turn the byte budget into a huge request list.
	MaxPrefetchChunks = 2048
)

Prefetch window bounds (#204). A file restore used to fetch every chunk on its own — with the S3 backend that is TWO presigned range requests per chunk (header, then payload), so a few hundred chunks meant a thousand round trips. Chunks are now fetched in windows: one pack-grouped, range-coalesced batch covers many chunks at once.

The window is what bounds memory. A restore holds AT MOST one window of raw frames at a time (the previous window is dropped before a new batch is issued), so the extra footprint is capped at MaxPrefetchBytes regardless of how large the file, the pack, or the backup is — nothing here buffers a whole file or a whole pack set. Streaming is otherwise unchanged: bytes are still written out chunk by chunk as the loop walks the entries.

View Source 
const (
	DigestMatch         = "match"
	DigestMismatch      = "mismatch"
	DigestNotVerifiable = "not-verifiable" // pre-#455 backup: no stored digest
)

Digest verdicts (#455): what a verify can honestly say about the manifest's whole-stream content digest. "" — no claim — is what a sampled or errored verify reports: a fold over part of the stream can neither match nor honestly mismatch the whole.

Variables

This section is empty.

Functions

func FilterFiles 

func FilterFiles(catalog []manifest.FileEntry, patterns []string) []manifest.FileEntry

filterFiles returns catalog entries matching any of the given patterns, plus parent directories needed to contain them.

func MatchFilePattern 

func MatchFilePattern(path, pattern string) bool

matchFilePattern checks if a file path matches a glob pattern.

func OpenEntryAccessor 

func OpenEntryAccessor(repoPath string, backup *manifest.Backup) (manifest.EntryAccessor, io.Closer)

openEntryAccessor opens an EntryAccessor for the given backup. It prefers the on-disk .dnm file for O(log n) seeks. If no disk file is available (e.g. in tests) it falls back to the already-loaded backup.Entries slice.

func OperatorExclusionFor 

func OperatorExclusionFor(excludePaths []string, catalogPath string) (string, bool)

OperatorExclusionFor finds the configured exclusion (canonical `C:\a\b` form) that covers a catalog path (forward-slash, volume-root relative, with or without a leading "./"). Case-insensitive, as NTFS paths are; a match is the path itself or anything under it.

func SampleEntryIndices 

func SampleEntryIndices(backup *manifest.Backup, percent float64, seed uint64) ([]int, error)

SampleEntryIndices deterministically selects a subset of a backup's verifiable (non-excluded) entry indices for --sample verification.

  • percent in (0,100]; percent >= 100 selects the whole verifiable population.
  • K = ceil(percent/100 * P), clamped to at least 1 (when there is anything to sample) and at most P.
  • Selection is seeded by the BackupID (optionally mixed with seed): the same (backup, percent, seed) yields the same subset on every host and rerun, so a flagged sample failure is reproducible.

The returned indices are TRUE manifest indices, sorted ascending (stable report order + better range coalescing).

func VerifyWrittenDigest 

func VerifyWrittenDigest(b *manifest.Backup, r io.Reader) (string, error)

VerifyWrittenDigest holds a restored target against the manifest's whole-stream digest (#455 slice 3): a sequential fold over exactly b.TotalBytes of r, compared to b.ContentDigest.

It reads the target back rather than folding at write time because the restorer writes PACK-MAJOR (#83) — every pack fetched exactly once, at the cost of stream order — so the only place the stream exists again is on the medium it was written to. That is also what makes this the STRONGEST check in the product: it judges the bytes on disk, after every buffer, cache and controller between the repo and the medium has had its say. Each chunk was verified as it was written; only this sees that what LANDED, as a whole, is what was captured.

Verdicts are the verify vocabulary: DigestMatch, DigestMismatch (a short read counts — truncation is the likeliest real failure, and "the image ended early" IS a mismatch with the captured stream, reported as one), DigestNotVerifiable for pre-digest backups. The error return is for the reader failing, not for the bytes disagreeing.

Types

type FileRestoreResult 

type FileRestoreResult struct {
	TotalFiles      int
	RestoredFiles   int
	DirsCreated     int
	SymlinksCreated int
	BytesWritten    int64
	Duration        time.Duration
}

FileRestoreResult contains the outcome of a file-mode restore.

type FileRestorer 

type FileRestorer struct {
	IgnoreErrors bool // if true, log and skip files with incomplete catalog entries rather than aborting
	// contains filtered or unexported fields
}

FileRestorer restores files from a file-mode backup.

func NewFileRestorer 

func NewFileRestorer(idx *index.DedupIndex, st *store.ChunkStore, repoPath string, logger *slog.Logger) *FileRestorer

NewFileRestorer creates a new file restore engine. repoPath is needed to load referenced manifests for unchanged files in watcher backups.

func (*FileRestorer) ExtractFile 

func (r *FileRestorer) ExtractFile(ctx context.Context, backup *manifest.Backup, filePath, outputPath string) (*FileRestoreResult, error)

ExtractFile extracts a single file from a file-mode backup to outputPath. Unlike RestoreFiles, this writes the file directly to outputPath without reconstructing the directory tree.

func (*FileRestorer) RestoreFiles 

func (r *FileRestorer) RestoreFiles(ctx context.Context, backup *manifest.Backup, targetDir string, filePatterns []string) (*FileRestoreResult, error)

RestoreFiles restores files from a file-mode backup to targetDir. If filePatterns is non-empty, only matching files (and their parent dirs) are restored.

func (*FileRestorer) SetNormalizer 

func (r *FileRestorer) SetNormalizer(n preprocess.Normalizer)

SetNormalizer configures the normalizer used to verify chunk integrity; it MUST match the normalizer the backup was created with. See Restorer.SetNormalizer.

type RestoreResult 

type RestoreResult struct {
	TotalChunks    int64
	RestoredChunks int64
	ExcludedChunks int64
	BytesWritten   int64
	Duration       time.Duration
}

RestoreResult contains the outcome of a restore operation.

type Restorer 

type Restorer struct {

	// OnProgress (#153): periodic (bytesWritten, totalBytes) during Restore —
	// recovery flows render percent/rate/ETA from it.
	OnProgress func(done, total int64)
	// contains filtered or unexported fields
}

Restorer restores backups from a repository to a target writer.

func NewRestorer 

func NewRestorer(idx *index.DedupIndex, st *store.ChunkStore, logger *slog.Logger) *Restorer

NewRestorer creates a new restore engine.

func (*Restorer) Restore 

func (r *Restorer) Restore(ctx context.Context, backup *manifest.Backup, writer Target) (*RestoreResult, error)

Restore writes the backup contents to the target writer. It is RestoreEntries over the backup's in-memory entries; a caller holding a DNM (or a chain of them) should pass its accessor to RestoreEntries directly and never hold the entries whole (#506).

func (*Restorer) RestoreEntries 

func (r *Restorer) RestoreEntries(ctx context.Context, backup *manifest.Backup, ea manifest.EntryAccessor, writer Target) (*RestoreResult, error)

RestoreEntries writes the backup to the target, reading its entries through ea instead of a materialized []Entry (#506). Two passes:

  1. stream the entries in batches: write zeros for excluded regions as they are met (one reused buffer, not one per region) and collect a chunkRef for every real chunk — the index lookup happens here;
  2. sort the refs pack-major (#83: every pack fetched exactly once) and restore each chunk, re-reading only that entry through ea.At, an O(1) seek on a DNM.

Restore is this with a slice accessor over backup.Entries; callers that hold a DNM (or a chain of them — an incremental's parents) pass its accessor and never hold the entries whole.

func (*Restorer) SetNormalizer 

func (r *Restorer) SetNormalizer(n preprocess.Normalizer)

SetNormalizer configures the normalizer used to verify chunk integrity. It MUST match the normalizer the backup was created with (reconstructed from the repo config): chunk identity is the hash of the normalized bytes while the stored bytes are the originals, so verification re-normalizes before comparing against the manifest's chunk hash. A nil normalizer (default) verifies the stored bytes directly.

type StreamVerify 

type StreamVerify struct {
	// contains filtered or unexported fields
}

StreamVerify is VerifyStreamed opened up for span-wise walking (#522 phase 3): the caller may verify the entry stream in consecutive ranges — staging each range's chunks under a disk budget between calls — and the digest fold carries across ranges, because SHA-256 state advanced in stream order is the SAME fold DigestCoversSourceStreamV1 names. Ranges MUST be walked consecutively from zero; Range enforces it, because a gap or overlap would silently fold a stream that is not the source.

func NewStreamVerify 

func NewStreamVerify(backup *manifest.Backup, total int64) (*StreamVerify, error)

NewStreamVerify prepares a walk over total entries of backup (METADATA only — Entries may be nil, exactly as VerifyStreamed).

func ResumeStreamVerify 

func ResumeStreamVerify(backup *manifest.Backup, total int64, checkpoint []byte) (*StreamVerify, error)

ResumeStreamVerify is NewStreamVerify continued from a Checkpoint. The next Range must start at the checkpoint's offset; Range enforces it.

func (*StreamVerify) Checkpoint 

func (sv *StreamVerify) Checkpoint() ([]byte, error)

Checkpoint serializes the walk's position so a later process can continue it (#522: a multi-span verify that outlives one run). It carries the entry offset, the counters, and the SHA-256 fold's own marshaled state — crypto/sha256 implements encoding.BinaryMarshaler, and a fold restored from it produces the digest an uninterrupted fold would, so DigestCoversSourceStreamV1 is untouched: still one sequential fold over the whole stream, across process lifetimes.

A walk that has seen a chunk error makes no stream claim and cannot be resumed: the fold is already aborted, and continuing would only spend hours to report a verdict Finish will refuse anyway.

func (*StreamVerify) Finish 

func (sv *StreamVerify) Finish() *VerifyResult

Finish closes the fold and returns the result. The digest verdict is only meaningful when every entry was ranged; Finish refuses a partial walk the same way the range check refuses a gap.

func (*StreamVerify) Next 

func (sv *StreamVerify) Next() int64

Next is the entry offset the next Range must start at.

func (*StreamVerify) Range 

func (sv *StreamVerify) Range(ctx context.Context, entries manifest.EntryAccessor, lo, hi int64, idx *index.DedupIndex, st *store.ChunkStore, norm preprocess.Normalizer, onProgress func(done, total int64)) error

Range verifies entries [lo, hi), folding them into the stream digest. onProgress reports GLOBAL positions against the full total.

type Target 

type Target interface {
	io.WriterAt
	Truncate(size int64) error
	Sync() error
}

Target is the destination a volume restore writes into. *volume.Writer satisfies it; the interface lives here so the core engine does not depend on the platform layer.

type VerifyError 

type VerifyError struct {
	ChunkIndex int
	Offset     int64
	Message    string
}

VerifyError describes a single verification failure.

func (VerifyError) Error 

func (e VerifyError) Error() string

type VerifyResult 

type VerifyResult struct {
	TotalChunks    int64
	VerifiedChunks int64
	ExcludedChunks int64
	Errors         []VerifyError
	Duration       time.Duration

	// DigestVerdict is the whole-stream check (#455): DigestMatch,
	// DigestMismatch, DigestNotVerifiable, or "" when this verify ran over
	// a sample (or hit chunk errors) and therefore makes no stream claim.
	// Per-chunk checks prove each chunk matches ITSELF; an entry list that
	// lost, duplicated or reordered a record passes every one of them —
	// only the fold over the reconstruction can object (#376, one level up).
	DigestVerdict string
	// DigestExpected/DigestActual are set on a mismatch, so the report
	// names both values instead of "they differ".
	DigestExpected string
	DigestActual   string
}

VerifyResult contains the outcome of a verify operation.

func Verify 

func Verify(ctx context.Context, backup *manifest.Backup, idx *index.DedupIndex, st *store.ChunkStore) (*VerifyResult, error)

Verify checks that all chunks in a backup are retrievable and match their hashes. Unlike Restore, it reports ALL errors instead of stopping at the first one.

Use VerifyWithNormalizer for repos created with --normalize; a nil normalizer (this function) verifies the stored bytes directly.

func VerifySelectedWithNormalizer 

func VerifySelectedWithNormalizer(ctx context.Context, backup *manifest.Backup, idx *index.DedupIndex, st *store.ChunkStore, norm preprocess.Normalizer, indices []int) (*VerifyResult, error)

func VerifyStreamed 

func VerifyStreamed(ctx context.Context, backup *manifest.Backup, entries manifest.EntryAccessor, idx *index.DedupIndex, st *store.ChunkStore, norm preprocess.Normalizer, onProgress func(done, total int64)) (*VerifyResult, error)

VerifySelectedWithNormalizer verifies only the entries at the given manifest indices (used by sampled cloud verify). Each VerifyError's ChunkIndex is the TRUE manifest index, so the report is unambiguous regardless of sampling. VerifyStreamed is the full verify over an EntryAccessor (#478): the walk restore-zip's memory lesson (#419) demands — windowed reads off the staged .dnm, never the whole entry list resident. It is by construction the complete in-order walk, so the digest fold applies exactly as in the slice path. backup supplies METADATA only (digest fields, the zero-entries guard); its Entries slice is ignored and may be nil. onProgress, when non-nil, is called every progressStride entries and at each window boundary with (verified so far, total) — the panel's percent for a run that takes minutes. Finer than the read window on purpose (the a00b2ce7 incident: one slow window froze progress for 11 minutes, and a stuck verify was indistinguishable from a slow one).

func VerifyWithNormalizer 

func VerifyWithNormalizer(ctx context.Context, backup *manifest.Backup, idx *index.DedupIndex, st *store.ChunkStore, norm preprocess.Normalizer) (*VerifyResult, error)

VerifyWithNormalizer is Verify for repos that used a normalizer. Chunk identity is the hash of the normalized bytes while the stored bytes are the originals, so the retrieved bytes are re-normalized before comparing against the manifest's chunk hash. norm must match the repo config.

func (*VerifyResult) OK 

func (r *VerifyResult) OK() bool

OK returns true if the backup verified without errors.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.