storage

package
v1.13.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 28, 2026 License: Apache-2.0 Imports: 40 Imported by: 0

Documentation

Overview

Package storage implements Coldkeep repository read/write semantics.

Frozen transform ordering contract (v1.9):

Canonical implementation entrypoints (v1.9):

  • Write path transform application: applyPackedBlockTransforms (store.go)
  • Read/verify staged inverse pipeline: verify.VerifyStoredBlock

Write path:

  1. logical encode
  2. logical hash
  3. compression
  4. compressed hash
  5. encryption
  6. physical hash
  7. persist

Read path (strict inverse):

  1. read payload
  2. physical hash verify
  3. decrypt
  4. compressed hash verify
  5. decompress
  6. logical hash verify
  7. decode logical block

This ordering is stable repository semantics. Repository defaults govern future writes only; reads are driven by persisted per-block metadata.

Frozen compression semantics (v1.9):

  • Scope: block-level only.
  • Timing: compression always runs before encryption.
  • Policy: store-if-smaller is canonical.
  • Reads: per-block metadata controls decompression behavior.

Compression semantics are part of repository compatibility and future engine contracts.

Frozen hash layer semantics (v1.9):

  • block_hash: canonical logical block identity.
  • payload_hash: deprecated lowercase-hex mirror of block_hash for compatibility/observability only.
  • compressed_hash: transform-stage integrity checkpoint.
  • physical_hash: persisted payload integrity checkpoint.

Identity authority is intentionally limited:

  • Only block_hash participates in logical correctness and restore identity.
  • payload_hash must not be used as a source of truth for identity.
  • Dedup, GC identity, snapshot identity, and restore graph semantics must not be derived from compressed_hash or physical_hash.

Index

Constants

View Source 
const (
	TestStoreInterleavingEventAfterChunkClaim            = storeInterleavingEventAfterChunkClaim
	TestStoreInterleavingEventBeforePackedFlush          = storeInterleavingEventBeforePackedFlush
	TestStoreInterleavingEventAfterPackedMetadata        = storeInterleavingEventAfterPackedMetadata
	TestStoreInterleavingEventAfterChunkCompleted        = storeInterleavingEventAfterChunkCompleted
	TestStoreInterleavingEventAfterLegacyCompanionInsert = storeInterleavingEventAfterLegacyCompanionInsert
	TestStoreInterleavingEventBeforePackedCommit         = storeInterleavingEventBeforePackedCommit
	TestStoreInterleavingEventAfterPackedCommit          = storeInterleavingEventAfterPackedCommit
	TestStoreInterleavingEventBeforeChunkRetryCAS        = storeInterleavingEventBeforeChunkRetryCAS
	TestStoreInterleavingEventBeforeMarkChunkForRebuild  = storeInterleavingEventBeforeMarkChunkForRebuild
	TestStoreInterleavingEventAfterMarkChunkForRebuild   = storeInterleavingEventAfterMarkChunkForRebuild
)

Variables

View Source 
var (
	ErrPhysicalPayloadHashMismatch   = errors.New("physical payload hash mismatch")
	ErrCompressedPayloadHashMismatch = errors.New("compressed payload hash mismatch")
	ErrLogicalBlockHashMismatch      = errors.New("logical block hash mismatch")
)

Functions

func ConfigureRestoreTestHooksForTesting added in v1.13.14 

func ConfigureRestoreTestHooksForTesting(
	s *StorageContext,
	beforeChunkRead func(*sql.DB, int64) error,
	failBeforeRename func(tempOutputPath, outputPath string) error,
)

ConfigureRestoreTestHooksForTesting installs deterministic restore hooks on one StorageContext. It is internal test infrastructure, not an application configuration surface. Configure the context before starting restore work.

func GetDefaultChunkerVersion added in v1.5.0 

func GetDefaultChunkerVersion(tx *sql.Tx) (chunk.Version, error)

GetDefaultChunkerVersion returns the repository-level default chunker version used for new writes.

Behavior contract: - if repository_config.default_chunker is absent, it returns v1-simple-rolling - returned values must be both well-formed and currently registered

func GetDefaultCompression added in v1.9.0 

func GetDefaultCompression(tx *sql.Tx) (string, error)

GetDefaultCompression returns the repository-level default compression codec.

Behavior contract: - if repository_config.compression is absent, it returns "none" - returned values must be registered/valid

func GetDefaultCompressionLevel added in v1.9.0 

func GetDefaultCompressionLevel(tx *sql.Tx) (int, error)

GetDefaultCompressionLevel returns the repository-level default compression level. Only valid when compression codec is "zstd".

Behavior contract: - if repository_config.compression_level is absent, it returns 3 - returned values must be in range [1, 9] for Phase 5.1 - level is only relevant when compression = "zstd"

func InstallTestStoreInterleavingHooks added in v1.13.9 

func InstallTestStoreInterleavingHooks(
	sgctx *StorageContext,
	onEvent func(context.Context, TestStoreInterleavingHookEvent) error,
) func()

func IsRegisteredCompressionCodec added in v1.9.0 

func IsRegisteredCompressionCodec(codec string) bool

IsRegisteredCompressionCodec returns true if the codec is valid for repository use. Supported compression codecs: "none" (passthrough), "zstd".

func LookupLogicalFileIDByStoredPath added in v1.13.8 

func LookupLogicalFileIDByStoredPath(dbconn *sql.DB, storedPath string) (int64, error)

LookupLogicalFileIDByStoredPath performs the current stored-path batch dry-run lookup without mutating catalog state. The caller owns trimming and duplicate handling semantics.

func LookupLogicalFileIDByStoredPathContext added in v1.13.14 

func LookupLogicalFileIDByStoredPathContext(ctx context.Context, dbconn *sql.DB, storedPath string) (int64, error)

LookupLogicalFileIDByStoredPathContext performs the lookup with caller-owned cancellation.

func RemoveFile 

func RemoveFile(fileID int64) error

func RemoveFileByStoredPathWithStorageContext added in v1.2.0 

func RemoveFileByStoredPathWithStorageContext(sgctx StorageContext, storedPath string) error

func RemoveFileWithDB 

func RemoveFileWithDB(dbconn *sql.DB, fileID int64) error

func RemoveFileWithDBContext added in v1.13.14 

func RemoveFileWithDBContext(ctx context.Context, dbconn *sql.DB, fileID int64) error

RemoveFileWithDBContext is the caller-context-aware form of RemoveFileWithDB.

func RestoreFile 

func RestoreFile(id int64, outputPath string) error

func RestoreFileByStoredPathWithStorageContext added in v1.2.0 

func RestoreFileByStoredPathWithStorageContext(sgctx StorageContext, storedPath string) error

func RestoreFileWithDB 

func RestoreFileWithDB(dbconn *sql.DB, fileID int64, outputPath string) error

func RestoreFileWithStorageContext added in v0.8.0 

func RestoreFileWithStorageContext(sgctx StorageContext, fileID int64, outputPath string) error

func SetDefaultChunkerVersion added in v1.5.0 

func SetDefaultChunkerVersion(tx *sql.Tx, v chunk.Version) error

SetDefaultChunkerVersion updates repository_config.default_chunker. The provided version must be well-formed and registered in the current binary.

func SetDefaultCompression added in v1.9.0 

func SetDefaultCompression(tx *sql.Tx, codec string) error

SetDefaultCompression updates repository_config.compression. The provided codec must be registered/valid.

func SetDefaultCompressionLevel added in v1.9.0 

func SetDefaultCompressionLevel(tx *sql.Tx, level int) error

SetDefaultCompressionLevel updates repository_config.compression_level. The provided level must be in range [1, 9] for Phase 5.1.

func StoreBlockPayload added in v0.7.0 

func StoreBlockPayload(c container.Container, payload []byte) (offset int64, newSize int64, err error)

Store payload bytes directly in a container and return offset/size metadata.

func StoreFile 

func StoreFile(path string) error

func StoreFileWithCodec added in v0.7.0 

func StoreFileWithCodec(path string, codec blocks.Codec) error

func StoreFileWithCodecString added in v0.8.0 

func StoreFileWithCodecString(path string, codecName string) error

func StoreFileWithStorageContext added in v0.8.0 

func StoreFileWithStorageContext(sgctx StorageContext, path string) (err error)

func StoreFileWithStorageContextAndCodec added in v0.8.0 

func StoreFileWithStorageContextAndCodec(sgctx StorageContext, path string, codec blocks.Codec) (err error)

func StoreFolder 

func StoreFolder(root string) error

func StoreFolderWithCodec added in v0.7.0 

func StoreFolderWithCodec(root string, codecName string) error

func StoreFolderWithStorageContext added in v0.8.0 

func StoreFolderWithStorageContext(sgctx StorageContext, root string) error

func StoreFolderWithStorageContextAndCodec added in v0.8.0 

func StoreFolderWithStorageContextAndCodec(sgctx StorageContext, root string, codec blocks.Codec) error

func StoreFolderWithStorageContextAndCodecAndOptions added in v1.7.0 

func StoreFolderWithStorageContextAndCodecAndOptions(sgctx StorageContext, root string, codec blocks.Codec, opts execution.Options) error

func StoreFolderWithStorageContextAndCodecAndOptionsWithStats added in v1.7.0 

func StoreFolderWithStorageContextAndCodecAndOptionsWithStats(sgctx StorageContext, root string, codec blocks.Codec, opts execution.Options) (execution.ExecutionStats, error)

func StoreFolderWithStorageContextAndCodecAndOptionsWithStatsContext added in v1.13.12 

func StoreFolderWithStorageContextAndCodecAndOptionsWithStatsContext(ctx context.Context, sgctx StorageContext, root string, codec blocks.Codec, opts execution.Options) (execution.ExecutionStats, error)

func StoreFolderWithStorageContextAndOptions added in v1.7.0 

func StoreFolderWithStorageContextAndOptions(sgctx StorageContext, root string, opts execution.Options) error

func ValidateRepositoryCompressionConfig added in v1.9.0 

func ValidateRepositoryCompressionConfig(tx *sql.Tx) error

ValidateRepositoryCompressionConfig validates the compression configuration during repository open/init.

Contract (Phase 5.1):

  • compression codec must be "none" or "zstd"
  • compression_level is only relevant for "zstd" codec
  • compression_level must be in range [1, 9] when set
  • missing compression config defaults to "none" (no compression)
  • missing compression_level defaults to 3 (when compression is not "none")
  • repository-level range [1, 9] is intentionally narrower than the compression library range [1, 22]

Types

type ActiveChunkerResolution added in v1.5.0 

type ActiveChunkerResolution struct {
	Chunker chunk.Chunker
	Version chunk.Version
}

ActiveChunkerResolution is the per-operation resolved chunker decision. It is resolved once and then reused for chunking and metadata persistence.

type BlockCache added in v1.8.0 

type BlockCache struct {
	// contains filtered or unexported fields
}

BlockCache is a per-restore in-memory cache for decoded blocks. Eviction policy: FIFO by first insertion order.

func (*BlockCache) Get added in v1.8.0 

func (c *BlockCache) Get(blockID int64) (*blocks.EncodedBlock, bool)

Get returns a cached block and whether it exists.

func (*BlockCache) Put added in v1.8.0 

func (c *BlockCache) Put(blockID int64, block *blocks.EncodedBlock)

Put inserts a block into the cache and evicts the oldest entry when full.

type BlockRequest added in v1.8.0 

type BlockRequest struct {
	// BlockID: identifier of the physical block to read
	BlockID int64
	// Segments: list of chunks that reside in this block
	// IMPORTANT: These are NOT in file output order; they're just grouped by block.
	// The restore loop maintains chunk_index order separately.
	Segments []*blocks.ChunkSegment
}

================================================================ Phase 3 Step 5: Block Grouping Types ================================================================ BlockRequest represents a physical block read with all chunks that reference it. Multiple chunks from the same file may reference the same block (possibly non-contiguously). Once a block is read and cached, all chunks referencing it can be sliced from the cached bytes.

type DualCompatChunkResolver added in v1.8.0 

type DualCompatChunkResolver struct {
	// contains filtered or unexported fields
}

DualCompatChunkResolver implements dual v1.7/v1.8 chunk resolution. It queries chunk_block_refs (v1.8) and falls back to legacy block mapping (v1.7).

v1.7 layout: each chunk stored as a single "block" (conceptually). 

storage_chunks.block_offset contains the chunk bytes.
Legacy: blockID ≡ chunkID (direct mapping).

v1.8 layout: chunks packed inside physical blocks. 

chunk_block_refs.(block_id, offset, size) define placement.
Advanced: many chunks → one physical block.

This resolver bridges both models seamlessly.

func NewDualCompatChunkResolver added in v1.8.0 

func NewDualCompatChunkResolver(db *sql.DB) *DualCompatChunkResolver

NewDualCompatChunkResolver creates a resolver supporting both v1.7 and v1.8 layouts.

func (*DualCompatChunkResolver) ResolveChunk added in v1.8.0 

func (r *DualCompatChunkResolver) ResolveChunk(ctx context.Context, chunkID int64) (*blocks.ChunkSegment, error)

ResolveChunk implements blocks.ChunkResolver. Returns ChunkSegment by checking v1.8 first, then falling back to v1.7.

type FileJob added in v1.7.0 

type FileJob struct {
	Index int
	Path  string
}

type HashMismatchError added in v1.9.0 

type HashMismatchError struct {
	Layer       hashLayer
	BlockID     int64
	ContainerID int64
	Offset      int64
	Expected    string
	Actual      string
}

HashMismatchError is a typed, layer-specific hash mismatch with safe context. It includes identifiers and digests only (never raw payload bytes or key material).

func (*HashMismatchError) Error added in v1.9.0 

func (e *HashMismatchError) Error() string

func (*HashMismatchError) Unwrap added in v1.9.0 

func (e *HashMismatchError) Unwrap() error

type LegacyBlockChunkResolver added in v1.8.0 

type LegacyBlockChunkResolver struct{}

LegacyBlockChunkResolver returns the v1.7 marker for all chunks (no DB queries). Used for v1.7-only repositories with no chunk_block_refs table.

func (*LegacyBlockChunkResolver) ResolveChunk added in v1.8.0 

func (r *LegacyBlockChunkResolver) ResolveChunk(ctx context.Context, chunkID int64) (*blocks.ChunkSegment, error)

ResolveChunk returns the v1.7 marker (BlockID == 0) signaling legacy direct-chunk path.

type LogicalFileInfo added in v1.1.0 

type LogicalFileInfo struct {
	FileID         int64
	OriginalName   string
	Status         string
	ChunkerVersion chunk.Version
}

LogicalFileInfo is lightweight metadata used by batch planning.

func GetLogicalFileInfoWithDB added in v1.1.0 

func GetLogicalFileInfoWithDB(dbconn *sql.DB, fileID int64) (LogicalFileInfo, error)

GetLogicalFileInfoWithDB returns logical file metadata for a given ID.

func GetLogicalFileInfoWithDBContext added in v1.13.14 

func GetLogicalFileInfoWithDBContext(ctx context.Context, dbconn *sql.DB, fileID int64) (LogicalFileInfo, error)

GetLogicalFileInfoWithDBContext returns logical file metadata using the caller-owned operation context.

type LogicalFileInspectInfo added in v1.5.0 

type LogicalFileInspectInfo struct {
	FileID            int64
	OriginalName      string
	Status            string
	ChunkerVersion    chunk.Version
	ChunkCount        int64
	AvgChunkSizeBytes float64
}

LogicalFileInspectInfo includes inspect-focused metadata for one logical file.

func GetLogicalFileInspectInfoWithDB added in v1.5.0 

func GetLogicalFileInspectInfoWithDB(dbconn *sql.DB, fileID int64) (LogicalFileInspectInfo, error)

GetLogicalFileInspectInfoWithDB returns inspect-focused metadata for a given file ID.

func GetLogicalFileInspectInfoWithDBContext added in v1.6.0 

func GetLogicalFileInspectInfoWithDBContext(ctx context.Context, dbconn *sql.DB, fileID int64) (LogicalFileInspectInfo, error)

GetLogicalFileInspectInfoWithDBContext returns inspect-focused metadata for a given file ID using the provided context.

type RemoveFileResult added in v0.8.0 

type RemoveFileResult struct {
	FileID          int64 `json:"file_id"`
	RemovedMappings int   `json:"removed_mappings"`
}

RemoveFileResult contains structured metadata about a remove operation. Remove is a state-changing path: it deletes logical-file mappings and decrements chunk live_ref_count values.

func RemoveFileWithDBResult added in v0.8.0 

func RemoveFileWithDBResult(dbconn *sql.DB, fileID int64) (result RemoveFileResult, err error)

func RemoveFileWithDBResultContext added in v1.13.14 

func RemoveFileWithDBResultContext(parent context.Context, dbconn *sql.DB, fileID int64) (result RemoveFileResult, err error)

RemoveFileWithDBResultContext preserves caller cancellation through the remove-by-ID transaction while retaining the contextless compatibility API.

type RemovePhysicalFileResult added in v1.2.0 

type RemovePhysicalFileResult struct {
	StoredPath        string `json:"stored_path"`
	LogicalFileID     int64  `json:"logical_file_id"`
	RemainingRefCount int64  `json:"remaining_ref_count"`
	Removed           bool   `json:"removed"`
}

RemovePhysicalFileResult contains structured metadata about unlinking a current-state physical_file mapping by path.

func RemoveFileByStoredPathWithStorageContextResult added in v1.2.0 

func RemoveFileByStoredPathWithStorageContextResult(sgctx StorageContext, storedPath string) (RemovePhysicalFileResult, error)

func RemoveFileByStoredPathWithStorageContextResultContext added in v1.13.14 

func RemoveFileByStoredPathWithStorageContextResultContext(ctx context.Context, sgctx StorageContext, storedPath string) (RemovePhysicalFileResult, error)

RemoveFileByStoredPathWithStorageContextResultContext preserves caller cancellation through the current-state unlink transaction.

type Repository added in v1.5.0 

type Repository struct {
	// contains filtered or unexported fields
}

Repository is the storage-layer persistence handle used by StoreService. It is the single point of DB access for store-path repository operations, keeping the service shape stable across the v1.9 frozen storage contract.

func NewRepository added in v1.5.0 

func NewRepository(db *sql.DB) *Repository

func (*Repository) DB added in v1.5.0 

func (r *Repository) DB() *sql.DB

func (*Repository) GetDefaultChunkerVersion added in v1.5.0 

func (r *Repository) GetDefaultChunkerVersion() (chunk.Version, error)

GetDefaultChunkerVersion returns the repository-level write default. It is transaction-backed so reads share the same persistence contract as other storage metadata accessors.

func (*Repository) GetDefaultCompression added in v1.9.0 

func (r *Repository) GetDefaultCompression(dbconn *sql.DB) (string, error)

GetDefaultCompression returns the repository-level default compression codec. The provided db handle is optional and allows callers to pass an explicit connection; when nil, the repository's configured DB is used.

func (*Repository) GetDefaultCompressionLevel added in v1.9.0 

func (r *Repository) GetDefaultCompressionLevel(dbconn *sql.DB) (int, error)

GetDefaultCompressionLevel returns the repository-level default compression level. The provided db handle is optional and allows callers to pass an explicit connection; when nil, the repository's configured DB is used.

func (*Repository) SetDefaultChunkerVersion added in v1.5.0 

func (r *Repository) SetDefaultChunkerVersion(v chunk.Version) error

SetDefaultChunkerVersion persists the repository-level write default.

func (*Repository) SetDefaultCompression added in v1.9.0 

func (r *Repository) SetDefaultCompression(dbconn *sql.DB, codec string) error

SetDefaultCompression persists the repository-level default compression codec. The provided db handle is optional and allows callers to pass an explicit connection; when nil, the repository's configured DB is used.

func (*Repository) SetDefaultCompressionLevel added in v1.9.0 

func (r *Repository) SetDefaultCompressionLevel(dbconn *sql.DB, level int) error

SetDefaultCompressionLevel persists the repository-level default compression level. The provided db handle is optional and allows callers to pass an explicit connection; when nil, the repository's configured DB is used.

type RepositoryCapabilities added in v1.9.0 

type RepositoryCapabilities = repositorycaps.RepositoryCapabilities

RepositoryCapabilities is the internal capability surface used by storage and upcoming engine extraction integration points.

func GetRepositoryCapabilities added in v1.9.0 

func GetRepositoryCapabilities(repo *Repository) RepositoryCapabilities

GetRepositoryCapabilities centralizes repository capability introspection for internal callers. Errors degrade to safe defaults so internal call sites can remain simple during extraction refactors.

func GetRepositoryCapabilitiesWithError added in v1.9.0 

func GetRepositoryCapabilitiesWithError(repo *Repository) (RepositoryCapabilities, error)

GetRepositoryCapabilitiesWithError is the strict variant for internal call sites that want introspection failures surfaced explicitly.

type RestoreDescriptor added in v1.2.0 

type RestoreDescriptor struct {
	Path               string
	LogicalFileID      int64
	Mode               sql.NullInt64
	MTime              sql.NullTime
	UID                sql.NullInt64
	GID                sql.NullInt64
	IsMetadataComplete bool
}

RestoreDescriptor describes a current-state restore target resolved from physical_file. It is the stable restore input shape that v1.3 snapshot/history restore can also produce.

type RestoreDestinationMode added in v1.2.0 

type RestoreDestinationMode string
const (
	RestoreDestinationOriginal RestoreDestinationMode = "original"
	RestoreDestinationPrefix   RestoreDestinationMode = "prefix"
	RestoreDestinationOverride RestoreDestinationMode = "override"
)

type RestoreFileResult added in v0.8.0 

type RestoreFileResult struct {
	FileID           int64                    `json:"file_id"`
	OriginalName     string                   `json:"original_name"`
	OutputPath       string                   `json:"output_path"`
	RestoredHash     string                   `json:"restored_hash"`
	MetadataWarnings *RestoreMetadataWarnings `json:"-"`
}

RestoreFileResult contains structured metadata about a restore operation.

func RestoreFileByStoredPathWithStorageContextResult added in v1.2.0 

func RestoreFileByStoredPathWithStorageContextResult(sgctx StorageContext, storedPath string) (RestoreFileResult, error)

func RestoreFileByStoredPathWithStorageContextResultOptions added in v1.2.0 

func RestoreFileByStoredPathWithStorageContextResultOptions(sgctx StorageContext, storedPath string, opts RestoreOptions) (RestoreFileResult, error)

RestoreFileByStoredPathWithStorageContextResultOptions restores a file using the current-state physical_file path as identity (v1.2 model). This is original destination mode: output path is the stored physical path.

func RestoreFileByStoredPathWithStorageContextResultOptionsContext added in v1.13.14 

func RestoreFileByStoredPathWithStorageContextResultOptionsContext(ctx context.Context, sgctx StorageContext, storedPath string, opts RestoreOptions) (RestoreFileResult, error)

RestoreFileByStoredPathWithStorageContextResultOptionsContext preserves caller cancellation through stored-path resolution and restore execution.

func RestoreFileWithDBResult added in v0.8.0 

func RestoreFileWithDBResult(dbconn *sql.DB, fileID int64, outputPath string) (RestoreFileResult, error)

func RestoreFileWithStorageContextResult added in v0.8.0 

func RestoreFileWithStorageContextResult(sgctx StorageContext, fileID int64, outputPath string) (RestoreFileResult, error)

func RestoreFileWithStorageContextResultOptions added in v1.1.0 

func RestoreFileWithStorageContextResultOptions(sgctx StorageContext, fileID int64, outputPath string, opts RestoreOptions) (RestoreFileResult, error)

func RestoreFileWithStorageContextResultOptionsContext added in v1.13.14 

func RestoreFileWithStorageContextResultOptionsContext(ctx context.Context, sgctx StorageContext, fileID int64, outputPath string, opts RestoreOptions) (RestoreFileResult, error)

RestoreFileWithStorageContextResultOptionsContext preserves caller cancellation through restore planning, reads, decoding, and publication.

type RestoreMetadata added in v1.13.13 

type RestoreMetadata struct {
	Mode  sql.NullInt64
	MTime sql.NullTime
	UID   sql.NullInt64
	GID   sql.NullInt64
}

type RestoreMetadataWarning added in v1.13.13 

type RestoreMetadataWarning struct {
	Operation string
	Detail    string
}

type RestoreMetadataWarnings added in v1.13.13 

type RestoreMetadataWarnings struct {
	Items []RestoreMetadataWarning
}

type RestoreOptions added in v1.1.0 

type RestoreOptions struct {
	Overwrite       bool
	DestinationMode RestoreDestinationMode
	Destination     string
	TrustedRoot     string
	StrictMetadata  bool
	NoMetadata      bool
	Metadata        *RestoreMetadata
	// contains filtered or unexported fields
}

RestoreOptions controls restore-file behavior.

type RestoreService added in v1.8.0 

type RestoreService struct {
	// ChunkResolver determines where restore finds chunks (block-based or legacy).
	// For v1.7 compatibility, returns a marker with BlockID == 0.
	// For v1.8 reads, returns ChunkSegment with actual BlockID, Offset, Size.
	ChunkResolver blocks.ChunkResolver

	// BlockReader decodes blocks for v1.8 block-based reads.
	// Only invoked when ChunkResolver returns a non-zero BlockID.
	BlockReader blocks.BlockReader
}

RestoreService provides restore operations with pluggable chunk resolution. It handles both v1.7 (direct legacy chunks) and v1.8 (packed block-based) layouts.

func NewDualCompatRestoreService added in v1.8.0 

func NewDualCompatRestoreService(db *sql.DB, containersDir string) *RestoreService

NewDualCompatRestoreService creates a restore service for mixed v1.7+v1.8 repositories. It uses a DualCompatChunkResolver to automatically detect and handle both layouts, and a StorageBlockReader to read v1.8 packed blocks from disk. containersDir must be the directory where container files are stored.

func NewLegacyRestoreService added in v1.8.0 

func NewLegacyRestoreService() *RestoreService

NewLegacyRestoreService creates a restore service for v1.7-only repositories. It uses LegacyBlockChunkResolver which always returns v1.7 markers.

func NewV17CompatRestoreService added in v1.8.0 

func NewV17CompatRestoreService() *RestoreService

NewV17CompatRestoreService creates a restore service for v1.7-only layouts. ChunkResolver returns the v1.7 marker; BlockReader is unused.

func (*RestoreService) InspectChunkResolution added in v1.8.0 

func (s *RestoreService) InspectChunkResolution(ctx context.Context, chunkID int64)

InspectChunkResolution logs the resolution path for debugging Phase 3 migrations.

func (*RestoreService) ReadChunkFromBlock added in v1.8.0 

func (s *RestoreService) ReadChunkFromBlock(ctx context.Context, blockID int64, offset, size int64) ([]byte, error)

ReadChunkFromBlock fetches a chunk payload from a v1.8 packed block using BlockReader. Called when ChunkSegment.BlockID > 0 (v1.8 block-based layout).

func (*RestoreService) ResolveChunkLocation added in v1.8.0 

func (s *RestoreService) ResolveChunkLocation(ctx context.Context, chunkID int64) (*blocks.ChunkSegment, error)

ResolveChunkLocation returns the physical location of a chunk. For v1.7 (BlockID == 0), the caller uses legacy storage_chunks table. For v1.8 (BlockID > 0), the caller fetches the block and slices the chunk.

type StorageBlockReader added in v1.8.0 

type StorageBlockReader struct {
	// contains filtered or unexported fields
}

StorageBlockReader implements blocks.BlockReader for reading blocks from storage. The read path mirrors the write pipeline in reverse: 

read payload → reverse transforms → verify logical hash → decode block

Layer semantics (inverse of write path): 

Layer 3 (persisted payload): raw bytes read from the container file.
Layer 2 (transformed payload): output of reverseTransforms (e.g. decrypted).
Layer 1 (logical block): plaintext encoded block bytes, verified against block_hash.

block_hash verification (Stage 4) is fail-closed: a mismatch or missing hash is always fatal. This anchors restore correctness to the logical layer regardless of which transforms are active.

v1.9 reverse transform behavior: 

verify physical_hash (if present) -> decrypt (if aes-gcm)
-> verify compressed_hash (if present) -> decompress (if zstd)
-> verify block_hash -> decode logical block.

func NewStorageBlockReader added in v1.8.0 

func NewStorageBlockReader(db *sql.DB, containersDir string) *StorageBlockReader

NewStorageBlockReader creates a BlockReader for storage_blocks-based block storage.

func (*StorageBlockReader) LogBlockRead added in v1.8.0 

func (r *StorageBlockReader) LogBlockRead(blockID int64, success bool, err error)

LogBlockRead logs block read operation for debugging.

func (*StorageBlockReader) ReadBlock added in v1.8.0 

func (r *StorageBlockReader) ReadBlock(ctx context.Context, blockID int64) (*blocks.EncodedBlock, error)

ReadBlock implements blocks.BlockReader.

Read pipeline (reverse of write):

  1. Load block metadata from storage_blocks
  2. Read stored bytes from container
  3. Reverse transforms (verify/decrypt/decompress per block metadata)
  4. Verify logical hash (mandatory, fail-closed)
  5. Decode block to reconstruct EncodedBlock

type StorageContext added in v0.8.0 

type StorageContext struct {
	DB           *sql.DB
	Writer       container.ContainerWriter
	ContainerDir string
	TempDBPath   string
	// Chunker overrides the chunking strategy used during store operations.
	// If nil, the registry default chunker is used. Set this in tests or when a
	// specific chunker version is required.
	Chunker chunk.Chunker
	// contains filtered or unexported fields
}

func LoadDefaultStorageContext added in v0.8.0 

func LoadDefaultStorageContext() (StorageContext, error)

LoadDefaultStorageContext resolves storage context from env with a secure default. Precedence: env (COLDKEEP_STORAGE_CONTEXT) -> default (local).

WARNING: this function does NOT run startup recovery. For the local backend, prefer OpenLocalStorage which includes recovery. Use this function only when you have already run recovery separately (e.g. the CLI startup path) or when operating in simulated mode.

func OpenLocalStorage added in v0.10.0 

func OpenLocalStorage(containersDir string) (StorageContext, error)

OpenLocalStorage is the recommended library entry point for the local storage backend. It runs startup recovery against containersDir first, then opens a ready StorageContext. Callers are guaranteed that in-progress logical files and chunks have been aborted, partially-sealed containers are resolved, and quarantine invariants hold before any store/restore operation begins.

If recovery fails the error is returned and no StorageContext is created. Callers that want fine-grained control over recovery (e.g. to emit a structured report) should call recovery.SystemRecoveryWithContainersDir themselves and then use ParseStorageContext or LoadDefaultStorageContext.

func ParseStorageContext added in v0.8.0 

func ParseStorageContext(value string) (StorageContext, error)

ParseStorageContext constructs a StorageContext for the given backend type. WARNING: this function does NOT run startup recovery. Callers are responsible for ensuring recovery.SystemRecoveryWithContainersDir has been called before performing any write operations, or use OpenLocalStorage which bundles both steps. ParseStorageContext is intentionally low-level to support simulated mode, testing, and advanced callers that manage the startup sequence themselves.

func (*StorageContext) Close added in v0.8.0 

func (s *StorageContext) Close() error

Close releases storage context resources. For simulated mode, it also removes the temporary sqlite DB file. Close is safe to call multiple times (idempotent). Writer finalization is always attempted first; in simulated mode this is a logical reset (no physical fsync/close), while local writers close handles. Note: some store paths also finalize writers per file operation; this call is intentionally tolerant as a defensive final ownership boundary.

func (StorageContext) EffectiveChunker added in v1.5.0 

func (s StorageContext) EffectiveChunker() chunk.Chunker

EffectiveChunker returns the configured Chunker or the default if none was set.

func (StorageContext) EffectiveContainerDir added in v0.8.0 

func (s StorageContext) EffectiveContainerDir() string

func (*StorageContext) IsSimulated added in v0.8.0 

func (s *StorageContext) IsSimulated() bool

type StorageContextType added in v0.8.0 

type StorageContextType string
const (
	LocalStorage     StorageContextType = "local"
	SimulatedStorage StorageContextType = "simulated"
	NasStorage       StorageContextType = "nas"
	S3Storage        StorageContextType = "s3"
)

type StoreFileResult added in v0.8.0 

type StoreFileResult struct {
	FileID        int64  `json:"file_id"`
	FileHash      string `json:"file_hash"`
	Path          string `json:"path"`
	AlreadyStored bool   `json:"already_stored"`
}

StoreFileResult contains structured metadata about a store operation. Store is a state-changing path: it mutates logical-file, chunk, block, and container state as payload is committed.

func StoreFileWithStorageContextAndCodecResult added in v0.8.0 

func StoreFileWithStorageContextAndCodecResult(sgctx StorageContext, path string, codec blocks.Codec) (result StoreFileResult, err error)

StoreFileWithStorageContextAndCodecResult stores one file and returns metadata suitable for CLI text and JSON output.

func StoreFileWithStorageContextAndCodecResultContext added in v1.13.14 

func StoreFileWithStorageContextAndCodecResultContext(ctx context.Context, sgctx StorageContext, path string, codec blocks.Codec) (result StoreFileResult, err error)

StoreFileWithStorageContextAndCodecResultContext stores one file while preserving caller cancellation through all ordinary work.

func StoreFileWithStorageContextAndCodecResultWithPolicy added in v1.2.0 

func StoreFileWithStorageContextAndCodecResultWithPolicy(sgctx StorageContext, path string, codec blocks.Codec, replace bool) (result StoreFileResult, err error)

StoreFileWithStorageContextAndCodecResultWithPolicy stores one file and returns metadata suitable for CLI text and JSON output, applying the given path-conflict policy. When replace is false, existing path mapped to different logical content fails. When replace is true, existing path mapping is atomically retargeted.

func StoreFileWithStorageContextAndCodecResultWithPolicyContext added in v1.13.14 

func StoreFileWithStorageContextAndCodecResultWithPolicyContext(ctx context.Context, sgctx StorageContext, path string, codec blocks.Codec, replace bool) (result StoreFileResult, err error)

StoreFileWithStorageContextAndCodecResultWithPolicyContext is the caller-context-aware form of StoreFileWithStorageContextAndCodecResultWithPolicy.

func StoreFileWithStorageContextResult added in v0.8.0 

func StoreFileWithStorageContextResult(sgctx StorageContext, path string) (StoreFileResult, error)

StoreFileWithStorageContextResult stores one file and returns structured result metadata.

func StoreFileWithStorageContextResultContext added in v1.13.14 

func StoreFileWithStorageContextResultContext(ctx context.Context, sgctx StorageContext, path string) (StoreFileResult, error)

StoreFileWithStorageContextResultContext is the caller-context-aware form of StoreFileWithStorageContextResult. Ordinary store work is owned by ctx.

type StoreService added in v1.5.0 

type StoreService struct {
	// contains filtered or unexported fields
}

StoreService owns store-path dependencies that must be resolved once per store operation. Phase 3 starts by making the active chunker explicit.

func NewStoreService added in v1.5.0 

func NewStoreService(repo *Repository, active chunk.Chunker) *StoreService

NewStoreService builds a store service.

If active is non-nil, it is treated as an explicit per-operation override. If active is nil, ResolveActiveChunker reads repository defaults once for the current operation.

func (*StoreService) ActiveChunker added in v1.5.0 

func (s *StoreService) ActiveChunker() chunk.Chunker

ActiveChunker returns the explicitly injected chunker override, if any.

func (*StoreService) Repository added in v1.5.0 

func (s *StoreService) Repository() *Repository

func (*StoreService) ResolveActiveChunker added in v1.5.0 

func (s *StoreService) ResolveActiveChunker() (ActiveChunkerResolution, error)

ResolveActiveChunker resolves and snapshots the active chunker decision for one store operation. Callers should invoke this once at operation start and reuse the returned Chunker and Version for the rest of the flow.

type TestStoreInterleavingEvent added in v1.13.9 

type TestStoreInterleavingEvent = storeInterleavingEvent

type TestStoreInterleavingHookEvent added in v1.13.9 

type TestStoreInterleavingHookEvent = storeInterleavingHookEvent

type WorkerStats added in v1.7.0 

type WorkerStats struct {
	Files int
	Bytes int64
}

Source Files

View all Source files

Directories

Path Synopsis
Package transforms defines composable transform primitives and a pipeline abstraction used for testability and v1.10 engine extraction preparation.
 Package transforms defines composable transform primitives and a pipeline abstraction used for testability and v1.10 engine extraction preparation.
aesgcm
Package aesgcm provides an AES-GCM encrypt/decrypt implementation of the transforms.Transform interface.
 Package aesgcm provides an AES-GCM encrypt/decrypt implementation of the transforms.Transform interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.