blocks

package
v1.13.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 28, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// CKBL in ASCII.
	BlockMagicV1 uint32 = 0x434B424C

	BlockFormatVersionV1 uint16 = 1
	BlockCodecNoneV1     uint16 = 0

	// BlockHashAlgorithm is the current project hash algorithm used for
	// plaintext encoded block hashing. v1.8 policy hashes encoded plaintext
	// block bytes (not encrypted bytes).
	BlockHashAlgorithm = "sha256"
)
View Source 
const (
	// DefaultPackedBlockTargetSizeBytes is the locked v1 packed-block target.
	DefaultPackedBlockTargetSizeBytes int64 = 1 << 20
)

Variables

View Source 
var (
	ErrBlockFormatTooSmall      = errors.New("block format: payload too small for header")
	ErrBlockFormatUnsupported   = errors.New("block format: unsupported header values")
	ErrBlockFormatInvalidLayout = errors.New("block format: invalid table/payload layout")
	ErrBlockFormatInvalidCount  = errors.New("block format: invalid chunk_count")
	ErrBlockFormatEmptyBlock    = errors.New("block format: empty block is not allowed")
	ErrBlockHashMismatch        = errors.New("block format: hash mismatch")
	ErrBlockHashExpectedNil     = errors.New("block format: expected hash must not be nil or empty")
	ErrNilEncodedBlock          = errors.New("block format: nil encoded block")
)
View Source 
var ErrBlockAlreadyExists = errors.New("block already exists for chunk")
View Source 
var ErrBlockBuilderCannotFit = errors.New("block builder: chunk does not fit current block")
View Source 
var ErrBlockBuilderChunkSizeMismatch = errors.New("block builder: chunk size does not match data length")
View Source 
var ErrBlockBuilderInvalidTargetSize = errors.New("block builder: invalid target size")
View Source 
var ErrBlockBuilderSizeOverflow = errors.New("block builder: size overflow")
View Source 
var ErrBlockBuilderZeroChunkSize = errors.New("block builder: zero-size chunk is not allowed")
View Source 
var V17ChunkSegmentMarker = &ChunkSegment{
	ChunkID: 0,
	BlockID: 0,
	Offset:  0,
	Size:    0,
}

V17ChunkSegmentMarker is a sentinel ChunkSegment indicating v1.7 direct chunk retrieval. BlockID == 0 and the other fields are zero-valued to signal legacy path.

Functions

func ComputeBlockHash added in v1.8.0 

func ComputeBlockHash(encoded []byte) []byte

ComputeBlockHash returns block hash over plaintext encoded block bytes.

IMPORTANT: hash target is encoded plaintext block bytes, before encryption. Delegates to HashLogical — the canonical hash helper for the logical layer.

func EncodeBlock added in v1.8.0 

func EncodeBlock(b *EncodedBlock) ([]byte, error)

EncodeBlock serializes one in-memory encoded block using deterministic v1 binary layout with little-endian fields: | HEADER | CHUNK_TABLE | PAYLOAD |

func GenerateKeyHex 

func GenerateKeyHex() (string, error)

func HashCompressed added in v1.9.0 

func HashCompressed(preEncryptionPayload []byte) []byte

HashCompressed returns the SHA-256 digest of the pre-encryption transform output (compressed bytes when compression is active, or encoded plaintext when compression is disabled).

When compression codec is "none": HashCompressed(x) == HashLogical(x).

func HashLogical added in v1.9.0 

func HashLogical(encodedPlaintext []byte) []byte

HashLogical returns the SHA-256 digest of the encoded plaintext block bytes. This is the canonical logical hash (block_hash) and is identical to the v1.8 block_hash contract. payload_hash is a deprecated lowercase-hex mirror retained for compatibility and observability only.

func HashPhysical added in v1.9.0 

func HashPhysical(persistedPayload []byte) []byte

HashPhysical returns the SHA-256 digest of the exact persisted payload bytes (i.e. after all transforms including encryption).

When no encryption is applied: HashPhysical(x) == HashCompressed(x).

func HashPlaintextEncodedBlock added in v1.8.0 

func HashPlaintextEncodedBlock(encoded []byte) []byte

HashPlaintextEncodedBlock remains as compatibility alias.

func LoadEncryptionKey 

func LoadEncryptionKey() ([]byte, error)

func SliceChunkFromPayload added in v1.8.0 

func SliceChunkFromPayload(payload []byte, entry ChunkEntry) ([]byte, error)

SliceChunkFromPayload returns chunk bytes for one table entry.

func VerifyBlockHash added in v1.8.0 

func VerifyBlockHash(encoded []byte, expected []byte) error

VerifyBlockHash verifies that the SHA-256 hash of encoded matches expected. Returns ErrBlockHashExpectedNil if expected is nil or empty. Returns ErrBlockHashMismatch (wrapping details) if hashes do not match.

Types

type AESGCMTransformer 

type AESGCMTransformer struct {
	Key []byte
}

func (*AESGCMTransformer) Decode 

func (t *AESGCMTransformer) Decode(_ context.Context, in DecodeInput) (data []byte, err error)

func (*AESGCMTransformer) Encode 

func (t *AESGCMTransformer) Encode(_ context.Context, in EncodeInput) (*TransformedBlock, error)

type Block added in v1.8.0 

type Block struct {
	ID              int64
	FormatVersion   int
	Codec           string
	Metadata        storagemetadata.BlockStorageMetadata
	ContainerID     int64
	ContainerOffset int64
}

Block represents a physical stored block unit for the v1.8 packed-block model. It captures the persisted block metadata shared by write, read, and verify paths.

type BlockBuilder added in v1.8.0 

type BlockBuilder struct {
	// contains filtered or unexported fields
}

BlockBuilder incrementally accumulates pending chunks for one packed block. Packing is deterministic: insertion order defines entry order.

func NewBlockBuilder added in v1.8.0 

func NewBlockBuilder(targetSize int64) *BlockBuilder

NewBlockBuilder creates a builder with the given target block size.

func (*BlockBuilder) Add added in v1.8.0 

func (b *BlockBuilder) Add(chunk PendingChunk) error

Add appends one pending chunk to the current block candidate.

func (*BlockBuilder) AddChunk added in v1.8.0 

func (b *BlockBuilder) AddChunk(id uint64, data []byte) error

AddChunk appends one chunk and updates aggregate plaintext size. Compatibility helper while write path migrates to PendingChunk API.

func (*BlockBuilder) Build added in v1.8.0 

func (b *BlockBuilder) Build() (*EncodedBlock, []byte, error)

Build constructs encoded block in-memory representation plus mandatory plaintext-encoded block hash.

func (*BlockBuilder) CanFit added in v1.8.0 

func (b *BlockBuilder) CanFit(size int64) bool

CanFit reports whether a chunk of given size can be added without splitting. Oversized chunks are allowed only when the builder is empty so they can be emitted alone after a caller flushes any current block.

func (*BlockBuilder) Empty added in v1.8.0 

func (b *BlockBuilder) Empty() bool

Empty reports whether there are no pending chunks.

func (*BlockBuilder) Reset added in v1.8.0 

func (b *BlockBuilder) Reset()

Reset clears current pending chunks so the builder can start a new block.

func (*BlockBuilder) ShouldFlushAtEnd added in v1.8.0 

func (b *BlockBuilder) ShouldFlushAtEnd() bool

ShouldFlushAtEnd applies deterministic end-of-operation flush rule: flush remaining pending chunks when operation ends.

func (*BlockBuilder) ShouldFlushBeforeAdd added in v1.8.0 

func (b *BlockBuilder) ShouldFlushBeforeAdd(nextSize int64) bool

ShouldFlushBeforeAdd applies deterministic flush rule checks before adding the next chunk. It returns true only for size-based reasons:

  1. current_size + next_size > target_size
  2. oversized next chunk must be written alone (when current block is non-empty)

It does not consider timing, goroutine completion, or random ordering.

type BlockHashes added in v1.9.0 

type BlockHashes struct {
	LogicalHash    []byte
	CompressedHash []byte
	PhysicalHash   []byte
}

BlockHashes holds hash digests for the three semantic layers of a stored block.

Layer semantics: 

LogicalHash    = hash(encoded plaintext block bytes) — "what was written logically"
                 Identical to the legacy block_hash (v1.8 contract preserved).
CompressedHash = hash(pre-encryption transform output) — "what the encryptor received"
                 When compression is disabled, CompressedHash == LogicalHash.
PhysicalHash   = hash(exact persisted payload bytes) — "what lives on disk / in the container"
                 When no encryption is applied, PhysicalHash == CompressedHash.

Phase 2 invariant (compression disabled, codec = "none"): 

CompressedHash == LogicalHash

type BlockHeader added in v1.8.0 

type BlockHeader struct {
	Magic         uint32
	Version       uint16
	Codec         uint16
	ChunkCount    uint32
	PlaintextSize uint64
}

BlockHeader is the fixed-size binary block header for v1 format.

Layout (20 bytes): - magic uint32 - version uint16 - codec uint16 - chunk_count uint32 - plaintext_size uint64

type BlockReader added in v1.8.0 

type BlockReader interface {
	ReadBlock(ctx context.Context, blockID int64) (*EncodedBlock, error)
}

BlockReader provides context-aware read access to decoded blocks. It supports both legacy and packed repository read flows.

type BlockStore added in v1.8.0 

type BlockStore interface {
	GetBlock(blockID int64) (*Block, error)
}

BlockStore is the storage-block retrieval boundary for packed layouts. The interface intentionally remains minimal to keep storage access decoupled.

type ChunkEntry added in v1.8.0 

type ChunkEntry struct {
	ChunkID uint64
	Offset  uint64
	Size    uint64
}

ChunkEntry is one chunk segment entry in the v1 chunk table.

Layout (24 bytes): - chunk_id uint64 - offset uint64 - size uint64

type ChunkLocator added in v1.8.0 

type ChunkLocator interface {
	GetChunkSegment(chunkID int64) (*ChunkSegment, error)
}

ChunkLocator resolves chunk placement inside a physical block. It isolates lookup logic from restore and verification execution paths.

type ChunkResolver added in v1.8.0 

type ChunkResolver interface {
	ResolveChunk(ctx context.Context, chunkID int64) (*ChunkSegment, error)
}

ChunkResolver provides context-aware resolution of chunk placement inside blocks. It supports chunk lookup across legacy and packed layouts.

type ChunkSegment added in v1.8.0 

type ChunkSegment struct {
	ChunkID int64
	BlockID int64
	Offset  int64
	Size    int64
}

ChunkSegment represents one chunk placement inside a physical block. Offset and Size are relative to decoded plaintext block bytes.

type Codec 

type Codec string
const (
	CodecPlain  Codec = "plain"
	CodecAESGCM Codec = "aes-gcm"
)

func LoadDefaultCodec 

func LoadDefaultCodec() (Codec, error)

LoadDefaultCodec resolves codec from env with a secure default. Precedence: env (COLDKEEP_CODEC) -> default (aes-gcm).

func ParseCodec 

func ParseCodec(value string) (Codec, error)

type DecodeInput 

type DecodeInput struct {
	ChunkHash  string
	Descriptor Descriptor
	Payload    []byte
}

type Descriptor 

type Descriptor struct {
	ID            int64
	ChunkID       int64
	Codec         Codec
	FormatVersion int
	PlaintextSize int64
	StoredSize    int64
	Nonce         []byte
	ContainerID   int64
	BlockOffset   int64
	CreatedAt     time.Time
	UpdatedAt     time.Time
}

Descriptor represents how a chunk is physically stored in the system. It links logical chunk identity to its encoded representation in a container.

type EncodeInput 

type EncodeInput struct {
	ChunkID   int64
	ChunkHash string
	Plaintext []byte
}

type EncodedBlock 

type EncodedBlock struct {
	Header   BlockHeader
	Entries  []ChunkEntry
	Payload  []byte
	Metadata TransformMetadata
}

EncodedBlock is the in-memory representation of a decoded/constructed plaintext block format payload in v1 layout.

func DecodeBlock added in v1.8.0 

func DecodeBlock(data []byte) (*EncodedBlock, error)

DecodeBlock parses and validates v1 encoded block bytes into in-memory representation.

func (*EncodedBlock) GetChunk added in v1.8.0 

func (b *EncodedBlock) GetChunk(i int) []byte

GetChunk returns one chunk slice by table index from the payload. Invalid indexes or invalid entry bounds return nil.

type EncodedPackedBlockV1 added in v1.8.0 

type EncodedPackedBlockV1 struct {
	Bytes     []byte
	BlockHash []byte
	Header    BlockHeader
	Entries   []ChunkEntry
}

EncodedPackedBlockV1 is the result of encoding v1 block bytes. BlockHash is mandatory and computed from plaintext encoded block bytes.

func EncodePackedBlockV1 added in v1.8.0 

func EncodePackedBlockV1(entries []ChunkEntry, payload []byte) (*EncodedPackedBlockV1, error)

EncodePackedBlockV1 encodes block bytes using deterministic v1 layout: | HEADER | CHUNK_TABLE | PAYLOAD |

func EncodePackedBlockV1FromChunks added in v1.8.0 

func EncodePackedBlockV1FromChunks(chunks []PackedChunk) (*EncodedPackedBlockV1, error)

EncodePackedBlockV1FromChunks builds a v1 encoded block from an ordered chunk sequence, creating chunk table segmentation and payload deterministically.

type PackedBlockTargetResolution added in v1.13.14 

type PackedBlockTargetResolution struct {
	Bytes       int64
	Megabytes   int64
	Warning     PackedBlockTargetWarning
	Environment string
}

PackedBlockTargetResolution is the pure effective target and warning data shared by Store and Stats.

func ResolvePackedBlockTarget added in v1.13.14 

func ResolvePackedBlockTarget() PackedBlockTargetResolution

ResolvePackedBlockTarget preserves the v1 Store environment precedence, parsing, supported-size set, and fallback semantics without producing logs.

type PackedBlockTargetWarning added in v1.13.14 

type PackedBlockTargetWarning uint8

PackedBlockTargetWarning classifies Store-compatible fallback warnings. 

const (
	PackedBlockTargetWarningNone PackedBlockTargetWarning = iota
	PackedBlockTargetWarningInvalid
	PackedBlockTargetWarningUnsupported
	PackedBlockTargetWarningOverflow
)

type PackedBlockV1 added in v1.8.0 

type PackedBlockV1 = EncodedBlock

PackedBlockV1 remains as a compatibility alias during Phase 2 rollout.

func DecodePackedBlockV1 added in v1.8.0 

func DecodePackedBlockV1(encoded []byte) (*PackedBlockV1, error)

DecodePackedBlockV1 parses and validates encoded v1 block bytes.

type PackedChunk added in v1.8.0 

type PackedChunk struct {
	ChunkID uint64
	Data    []byte
}

PackedChunk is an encode helper input for building payload and chunk table deterministically from an ordered chunk sequence.

type PendingChunk added in v1.8.0 

type PendingChunk struct {
	ChunkID int64
	Hash    []byte
	Data    []byte
	Size    int64
}

PendingChunk is one chunk candidate waiting to be packed into a block. Hash contains the chunk identity hash over plaintext bytes.

type PlainTransformer 

type PlainTransformer struct{}

PlainTransformer stores chunks as-is without any transformation.

func (*PlainTransformer) Decode 

func (t *PlainTransformer) Decode(_ context.Context, in DecodeInput) ([]byte, error)

No transformation needed for plain codec, just return the payload as-is. Phase 6 Step 8: Avoid unnecessary byte copies during restore - Decode returns payload directly without copying - Caller (restore.go) never mutates plaintext after verification - Direct return saves memory allocation and copy time per chunk

func (*PlainTransformer) Encode 

func (t *PlainTransformer) Encode(_ context.Context, in EncodeInput) (*TransformedBlock, error)

type Repository 

type Repository struct {
	DB *sql.DB
}

func (*Repository) GetByChunkHash 

func (r *Repository) GetByChunkHash(ctx context.Context, chunkHash string) (*Descriptor, error)

func (*Repository) GetByChunkID 

func (r *Repository) GetByChunkID(ctx context.Context, chunkID int64) (*Descriptor, error)

func (*Repository) Insert 

func (r *Repository) Insert(ctx context.Context, tx *sql.Tx, d *Descriptor) error

type StorageChunkResolver added in v1.8.0 

type StorageChunkResolver struct {
	// contains filtered or unexported fields
}

StorageChunkResolver implements ChunkResolver for v1.8 block-based layout. It resolves chunk locations by querying the chunk_block_refs and storage_blocks tables.

func NewStorageChunkResolver added in v1.8.0 

func NewStorageChunkResolver(locator ChunkLocator) *StorageChunkResolver

NewStorageChunkResolver creates a chunk resolver backed by the given ChunkLocator.

func (*StorageChunkResolver) ResolveChunk added in v1.8.0 

func (r *StorageChunkResolver) ResolveChunk(ctx context.Context, chunkID int64) (*ChunkSegment, error)

ResolveChunk looks up which block contains the chunk and returns the segment location.

type TransformMetadata added in v1.9.0 

type TransformMetadata struct {
	// PayloadHash is a compatibility/observability mirror of the logical block hash
	// (lowercase-hex SHA256 of encoded plaintext block bytes before transforms).
	// storage_blocks.block_hash remains the authoritative logical identity.
	PayloadHash string

	// CompressionCodec identifies the compression stage outcome persisted per block.
	// v1.9 values are 'none' and 'zstd'. Store-if-smaller may keep 'none' even when
	// zstd is configured for the repository.
	CompressionCodec string

	// CompressionRatio is compressed_payload_size / encoded_plaintext_size
	// for the pre-encryption compression stage.
	// 1.0 indicates no effective compression (including store-if-smaller fallback),
	// values < 1.0 indicate successful size reduction.
	CompressionRatio float64
}

TransformMetadata carries explicit transformation information through the persistence pipeline. It threads compression metadata, payload information, and hash data from encode stage through transform stage to final persistence for the current v1.9 write/read contract.

type TransformedBlock added in v1.8.0 

type TransformedBlock struct {
	Descriptor Descriptor
	Payload    []byte
}

type Transformer 

type Transformer interface {
	Encode(ctx context.Context, in EncodeInput) (*TransformedBlock, error)
	Decode(ctx context.Context, in DecodeInput) ([]byte, error)
}

Transformer defines how a chunk is transformed into a stored block and how a stored block is transformed back into plaintext.

func GetBlockTransformer 

func GetBlockTransformer(codec Codec) (Transformer, error)

get codec transformer from codec name

type V17CompatResolver added in v1.8.0 

type V17CompatResolver struct{}

V17CompatResolver is a no-op resolver for v1.7 layouts. It signals that chunk resolution must fall back to legacy storage_chunks table logic. In the restore path, when ChunkResolver returns V17ChunkSegmentMarker, the system uses pre-v1.8 direct-chunk retrieval instead of block-based lookup.

func (*V17CompatResolver) ResolveChunk added in v1.8.0 

func (r *V17CompatResolver) ResolveChunk(ctx context.Context, chunkID int64) (*ChunkSegment, error)

ResolveChunk returns the v1.7 marker, signaling direct chunk lookup without block read.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.