engine

package
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 27, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var DebugWriter io.Writer

DebugWriter, when non-nil, receives debug output from engine internals (cache stats, index reconciliation, cache misses). Set from main when the -debug flag is active.

Functions

func AddFileMetasToIndex 

func AddFileMetasToIndex(s store.ObjectStore, metas map[string]core.FileMeta) error

AddFileMetasToIndex merges a batch of file-meta entries into the catalog. It is safe to call even when the catalog does not yet exist.

func AddSnapshotToIndex 

func AddSnapshotToIndex(s store.ObjectStore, snap core.Snapshot, ref string) error

AddSnapshotToIndex appends a snapshot to the catalog. It is safe to call even when the catalog does not yet exist.

func LoadFileMetaCache 

func LoadFileMetaCache(s store.ObjectStore) (map[string]core.FileMeta, error)

LoadFileMetaCache returns a ref→FileMeta map that is always consistent with the store. It lists all live filemeta/ keys and loads the cached catalog, then reconciles:

  • key exists + in cache → trust the cached value (zero cost)
  • key exists + not cached → fetch from store, add to cache
  • key gone + in cache → drop from cache

The updated cache is flushed back to the store when anything changed.

func LoadNodeCache 

func LoadNodeCache(s store.ObjectStore) map[string][]byte

LoadNodeCache reads the persisted node catalog from the store. Returns an empty map (not nil) when the catalog does not yet exist.

func NewCachedTree 

func NewCachedTree(s store.ObjectStore) *hamt.Tree

NewCachedTree creates a HAMT tree backed by a TransactionalStore whose read cache is pre-populated from the persistent node catalog. This avoids individual B2 fetches for nodes that were seen in previous runs.

func RemoveSnapshotFromIndex 

func RemoveSnapshotFromIndex(s store.ObjectStore, ref string) error

RemoveSnapshotFromIndex removes a snapshot from the catalog. It is safe to call even when the ref is not present.

func SaveContentCache 

func SaveContentCache(s store.ObjectStore, cat ContentCatalog) error

func SaveNodeCache 

func SaveNodeCache(s store.ObjectStore, nodes map[string][]byte) error

SaveNodeCache persists all node data so the next run can skip B2 fetches.

Types

type BackupManager 

type BackupManager struct {
	// contains filtered or unexported fields
}

BackupManager orchestrates a backup: scanning a source for changes, uploading new or modified files, and persisting a snapshot backed by a Merkle-HAMT.

func NewBackupManager 

func NewBackupManager(src store.Source, dest store.ObjectStore, reporter ui.Reporter, opts ...BackupOption) *BackupManager

func (*BackupManager) Run 

func (bm *BackupManager) Run(ctx context.Context) (*RunResult, error)

Run executes a full backup: scan the source for changes, upload new/modified files, build a new HAMT root, and persist a snapshot.

type BackupOption 

type BackupOption func(*backupConfig)

BackupOption configures a backup operation.

func WithBackupDryRun 

func WithBackupDryRun() BackupOption

WithBackupDryRun scans the source and reports what would change without writing to the store.

func WithGenerator 

func WithGenerator(name string) BackupOption

WithGenerator overrides the default generator name in snapshot metadata.

func WithMeta 

func WithMeta(key, value string) BackupOption

WithMeta adds a key-value pair to the snapshot metadata.

func WithTags 

func WithTags(tags ...string) BackupOption

WithTags adds tags to the backup snapshot.

func WithVerbose 

func WithVerbose() BackupOption

WithVerbose enables verbose output during backup.

type ChangeType 

type ChangeType string

ChangeType describes how a file differs between two snapshots. 

const (
	ChangeAdded    ChangeType = "A"
	ChangeRemoved  ChangeType = "D"
	ChangeModified ChangeType = "M"
)

type Chunker 

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker splits a byte stream into content-defined chunks, deduplicates them, and persists the resulting Content object.

func NewChunker 

func NewChunker(s store.ObjectStore) *Chunker

func (*Chunker) CreateContentObject 

func (c *Chunker) CreateContentObject(chunkRefs []string, size int64, contentHash string) (string, error)

CreateContentObject persists a Content object keyed by contentHash and returns its store ref.

func (*Chunker) ProcessStream 

func (c *Chunker) ProcessStream(r io.Reader, onProgress func(int64)) (refs []string, size int64, hash string, err error)

ProcessStream splits r into content-defined chunks and stores each one (skipping duplicates). It returns the ordered chunk refs, total byte count, and the SHA-256 content hash over the raw stream.

onProgress is called after each chunk with the number of raw bytes consumed.

type ContentCatalog 

type ContentCatalog map[string][]string

ContentCatalog maps content refs ("content/<hash>") to their chunk refs.

func LoadContentCache 

func LoadContentCache(s store.ObjectStore) ContentCatalog

type DiffManager 

type DiffManager struct {
	// contains filtered or unexported fields
}

DiffManager compares two snapshots and reports file-level changes.

func NewDiffManager 

func NewDiffManager(s store.ObjectStore) *DiffManager

func (*DiffManager) Run 

func (dm *DiffManager) Run(ctx context.Context, snapID1, snapID2 string, opts ...DiffOption) (*DiffResult, error)

Run resolves two snapshot IDs and computes the diff.

type DiffOption 

type DiffOption func(*diffConfig)

DiffOption configures a diff operation.

type DiffResult 

type DiffResult struct {
	Ref1    string
	Ref2    string
	Changes []FileChange
}

DiffResult holds the outcome of a diff operation.

type FileChange 

type FileChange struct {
	Type ChangeType
	Path string
	Meta core.FileMeta
}

FileChange is a single entry in a diff report.

type ForgetManager 

type ForgetManager struct {
	// contains filtered or unexported fields
}

ForgetManager removes a snapshot and its index pointers, optionally pruning unreachable objects afterwards.

func NewForgetManager 

func NewForgetManager(s store.ObjectStore, reporter ui.Reporter) *ForgetManager

func (*ForgetManager) Run 

func (fm *ForgetManager) Run(ctx context.Context, snapshotID string, opts ...ForgetOption) (*ForgetResult, error)

Run removes the snapshot identified by snapshotID.

func (*ForgetManager) RunPolicy 

func (fm *ForgetManager) RunPolicy(ctx context.Context, opts ...ForgetOption) (*PolicyResult, error)

RunPolicy applies a retention policy to all snapshots and removes those not matched by any keep rule. Use WithKeepLast, WithKeepDaily, etc. to configure.

type ForgetOption 

type ForgetOption func(*forgetConfig)

ForgetOption configures a forget operation.

func WithDryRun 

func WithDryRun() ForgetOption

WithDryRun shows what would be removed without actually removing anything.

func WithFilterAccount 

func WithFilterAccount(account string) ForgetOption

WithFilterAccount restricts the policy to snapshots from this account.

func WithFilterPath 

func WithFilterPath(path string) ForgetOption

WithFilterPath restricts the policy to snapshots from this path.

func WithFilterSource 

func WithFilterSource(source string) ForgetOption

WithFilterSource restricts the policy to snapshots from this source type.

func WithFilterTag 

func WithFilterTag(tag string) ForgetOption

WithFilterTag restricts the policy to snapshots that have this tag.

func WithGroupBy 

func WithGroupBy(fields string) ForgetOption

WithGroupBy sets the fields used to group snapshots for policy application. Comma-separated list of: source, account, path, tags. Empty string disables grouping.

func WithKeepDaily 

func WithKeepDaily(n int) ForgetOption

WithKeepDaily keeps one snapshot per day for the last n days that have snapshots.

func WithKeepHourly 

func WithKeepHourly(n int) ForgetOption

WithKeepHourly keeps one snapshot per hour for the last n hours that have snapshots.

func WithKeepLast 

func WithKeepLast(n int) ForgetOption

WithKeepLast keeps the n most recent snapshots.

func WithKeepMonthly 

func WithKeepMonthly(n int) ForgetOption

WithKeepMonthly keeps one snapshot per month for the last n months that have snapshots.

func WithKeepWeekly 

func WithKeepWeekly(n int) ForgetOption

WithKeepWeekly keeps one snapshot per ISO week for the last n weeks that have snapshots.

func WithKeepYearly 

func WithKeepYearly(n int) ForgetOption

WithKeepYearly keeps one snapshot per year for the last n years that have snapshots.

func WithPrune 

func WithPrune() ForgetOption

WithPrune runs a prune pass after forgetting snapshots.

type ForgetPolicy 

type ForgetPolicy struct {
	KeepLast    int
	KeepHourly  int
	KeepDaily   int
	KeepWeekly  int
	KeepMonthly int
	KeepYearly  int
}

ForgetPolicy describes which snapshots to keep.

func (ForgetPolicy) IsEmpty 

func (p ForgetPolicy) IsEmpty() bool

func (ForgetPolicy) String 

func (p ForgetPolicy) String() string

type ForgetResult 

type ForgetResult struct {
	Prune *PruneResult // nil when prune was not requested
}

ForgetResult holds the outcome of a forget operation.

type GroupKey 

type GroupKey struct {
	Source  string
	Account string
	Path    string
	Tags    string // sorted, comma-joined
}

GroupKey identifies a group of snapshots for policy application.

func (GroupKey) String 

func (k GroupKey) String() string

type KeepReason 

type KeepReason struct {
	Entry   SnapshotEntry
	Reasons []string
}

KeepReason pairs a snapshot with the reasons it was kept.

type ListManager 

type ListManager struct {
	// contains filtered or unexported fields
}

ListManager enumerates all available snapshots.

func NewListManager 

func NewListManager(s store.ObjectStore) *ListManager

func (*ListManager) Run 

func (lm *ListManager) Run(ctx context.Context, opts ...ListOption) (*ListResult, error)

Run lists every snapshot in the store.

type ListOption 

type ListOption func(*listConfig)

ListOption configures a list operation.

type ListResult 

type ListResult struct {
	Snapshots []SnapshotEntry
}

ListResult holds the snapshots returned by a list operation.

type LsSnapshotManager 

type LsSnapshotManager struct {
	// contains filtered or unexported fields
}

LsSnapshotManager lists the file tree of a single snapshot.

func NewLsSnapshotManager 

func NewLsSnapshotManager(s store.ObjectStore) *LsSnapshotManager

func (*LsSnapshotManager) Run 

func (lm *LsSnapshotManager) Run(ctx context.Context, snapshotID string, opts ...LsSnapshotOption) (*LsSnapshotResult, error)

Run resolves the snapshot, collects metadata, and returns the tree structure.

type LsSnapshotOption 

type LsSnapshotOption func(*lsSnapshotConfig)

LsSnapshotOption configures an ls-snapshot operation.

type LsSnapshotResult 

type LsSnapshotResult struct {
	Ref       string
	Snapshot  core.Snapshot
	RootRefs  []string
	RefToMeta map[string]core.FileMeta
	ChildRefs map[string][]string
}

LsSnapshotResult holds the data returned by an ls-snapshot operation.

type NodeCatalog 

type NodeCatalog map[string]json.RawMessage

NodeCatalog maps node refs (e.g. "node/<hash>") to their raw JSON bytes.

type PolicyGroupResult 

type PolicyGroupResult struct {
	Key    GroupKey
	Keep   []KeepReason
	Remove []SnapshotEntry
}

PolicyGroupResult holds the policy evaluation result for a single group.

type PolicyResult 

type PolicyResult struct {
	Groups []PolicyGroupResult
	Prune  *PruneResult
}

PolicyResult holds the outcome of a policy-based forget operation.

type PruneManager 

type PruneManager struct {
	// contains filtered or unexported fields
}

PruneManager implements mark-and-sweep garbage collection over the object store.

func NewPruneManager 

func NewPruneManager(s store.ObjectStore, reporter ui.Reporter) *PruneManager

func (*PruneManager) Run 

func (pm *PruneManager) Run(ctx context.Context, opts ...PruneOption) (*PruneResult, error)

type PruneOption 

type PruneOption func(*pruneConfig)

func WithPruneDryRun 

func WithPruneDryRun() PruneOption

func WithPruneVerbose 

func WithPruneVerbose() PruneOption

type PruneResult 

type PruneResult struct {
	BytesReclaimed int64
	ObjectsDeleted int
	ObjectsScanned int
	DryRun         bool
}

type RestoreManager 

type RestoreManager struct {
	// contains filtered or unexported fields
}

RestoreManager recreates a snapshot's file tree as a ZIP archive.

func NewRestoreManager 

func NewRestoreManager(s store.ObjectStore, reporter ui.Reporter) *RestoreManager

func (*RestoreManager) Run 

func (rm *RestoreManager) Run(ctx context.Context, w io.Writer, snapshotRef string, opts ...RestoreOption) (*RestoreResult, error)

Run writes the snapshot's file tree as a ZIP archive to w. snapshotRef can be "", "latest", a bare hash, or "snapshot/<hash>".

type RestoreOption 

type RestoreOption func(*restoreConfig)

RestoreOption configures a restore operation.

func WithRestoreDryRun 

func WithRestoreDryRun() RestoreOption

WithRestoreDryRun resolves the snapshot and reports what would be restored without writing the archive.

func WithRestoreVerbose 

func WithRestoreVerbose() RestoreOption

WithRestoreVerbose logs each file/dir being written.

type RestoreResult 

type RestoreResult struct {
	SnapshotRef  string
	Root         string
	FilesWritten int
	DirsWritten  int
	BytesWritten int64
	Errors       int
	DryRun       bool
}

RestoreResult holds the outcome of a restore operation.

type RunResult 

type RunResult struct {
	SnapshotHash     string
	SnapshotRef      string
	Root             string
	FilesNew         int64
	FilesChanged     int64
	FilesUnmodified  int64
	FilesRemoved     int64
	DirsNew          int64
	DirsChanged      int64
	DirsUnmodified   int64
	DirsRemoved      int64
	BytesAddedRaw    int64
	BytesAddedStored int64
	Duration         time.Duration
	DryRun           bool
}

RunResult holds the outcome of a successful backup run.

type SnapshotEntry 

type SnapshotEntry struct {
	Ref     string
	Snap    core.Snapshot
	Created time.Time
}

SnapshotEntry is a snapshot loaded for policy evaluation.

func ListAllSnapshots 

func ListAllSnapshots(s store.ObjectStore) ([]SnapshotEntry, error)

ListAllSnapshots enumerates every snapshot in the store. It reconciles the index/snapshots catalog against the live snapshot/ key listing so the catalog is self-healing: missing entries are fetched from the store and stale entries are pruned. Results are sorted newest-first by Created time.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.