storage

package
v0.1.0-alpha.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 24, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package storage implements the current Meldbase copy-on-write page format.

Index

Constants

View Source 
const (
	DefaultCommitRetentionMaxCommits uint64 = 10_000
	DefaultCommitRetentionMaxBytes   uint64 = 256 << 20
	DefaultMaxFileBytes              uint64 = 8 << 30
)
View Source 
const (
	PageSize       = 16 * 1024
	PageHeaderSize = 64
	MetaHeaderSize = 256
	FormatVersion  = 3
)
View Source 
const (
	// Required feature bits must be understood before any generation is opened.
	RequiredFeatureShadowIndexBuilds uint64 = 1 << 0
	RequiredFeatureCompoundIndexes   uint64 = 1 << 1
	// RequiredFeatureIndexBuildAppliedRoot means catch-up build records protect
	// the exact CatalogRoot represented by their AppliedSequence watermark.
	RequiredFeatureIndexBuildAppliedRoot uint64 = 1 << 2
	SupportedRequiredFeatures            uint64 = RequiredFeatureShadowIndexBuilds | RequiredFeatureCompoundIndexes | RequiredFeatureIndexBuildAppliedRoot
	// Unknown optional bits are preserved and ignored. A writer may only alter
	// bits it owns; this permits older binaries to carry future acceleration
	// metadata through a normal commit without treating it as allocation truth.
	OptionalFeaturePersistentFreeSpace uint64 = 1 << 0
	SupportedOptionalFeatures          uint64 = OptionalFeaturePersistentFreeSpace
)
View Source 
const (
	MaxConcurrentIndexBuilds      = 64
	MaxIndexBuildBatchEntries     = 4096
	MaxIndexBuildBatchBytes       = 16 << 20
	MaxIndexBuildCatchUpCommits   = 1024
	MaxIndexBuildCatchUpMutations = 10_000
)
View Source 
const (
	MaxSecondaryScalarKeyBytes = 4096 - 24
)

Variables

View Source 
var (
	ErrNodeFull  = errors.New("meldbase storage: tree node is full")
	ErrKeyTooBig = errors.New("meldbase storage: tree key is too large")
)
View Source 
var (
	ErrHistoryLost       = errors.New("meldbase storage: commit history is no longer retained")
	ErrHistoryPinned     = errors.New("meldbase storage: commit history is pinned by a live replay cursor")
	ErrNoDeliveredCommit = errors.New("meldbase storage: live stream has not delivered a commit")
	ErrCursorClosed      = errors.New("meldbase storage: commit cursor is closed")
)
View Source 
var (

	// ErrDurableConsumerExists prevents an accidental reset of a durable
	// checkpoint. Callers must explicitly open the existing consumer instead.
	ErrDurableConsumerExists   = errors.New("meldbase storage: durable consumer already exists")
	ErrDurableConsumerNotFound = errors.New("meldbase storage: durable consumer not found")
)
View Source 
var (
	ErrLocked              = errors.New("meldbase storage: database is locked")
	ErrRecoveryRequired    = errors.New("meldbase storage: recovery required")
	ErrInvalidStorageLimit = errors.New("meldbase storage: invalid storage limit")
	ErrStorageLimit        = errors.New("meldbase storage: storage limit exceeded")
	ErrStaleSnapshot       = errors.New("meldbase storage: commit sequence is below the required minimum")
	ErrDatabaseIdentity    = errors.New("meldbase storage: unexpected database identity")
	ErrInsecureFileMode    = errors.New("meldbase storage: database file permissions are not owner-private")
)
View Source 
var (
	ErrCorrupt            = errors.New("meldbase storage: corrupt database")
	ErrUnsupportedFormat  = errors.New("meldbase storage: unsupported format version")
	ErrUnsupportedFeature = errors.New("meldbase storage: unsupported required feature")
)
View Source 
var (
	ErrIndexBuildExists      = errors.New("meldbase storage: index build exists")
	ErrIndexBuildNotFound    = errors.New("meldbase storage: index build not found")
	ErrIndexBuildState       = errors.New("meldbase storage: invalid index build state")
	ErrIndexBuildHistoryLost = errors.New("meldbase storage: index build commit history lost")
)
View Source 
var (
	ErrIndexExists      = errors.New("meldbase storage: index name exists")
	ErrIndexNotFound    = errors.New("meldbase storage: index name not found")
	ErrUniqueConflict   = errors.New("meldbase storage: unique index conflict")
	ErrIndexKeyTooLarge = errors.New("meldbase storage: index key is too large")
)
View Source 
var ErrCollectionExists = errors.New("meldbase storage: collection exists")
View Source 
var ErrDocumentConflict = errors.New("meldbase storage: document precondition conflicted")

ErrDocumentConflict reports that an optimistic point precondition no longer matches the current root. It is a safe logical rejection, not corruption or a durability failure.

View Source 
var ErrDocumentExists = errors.New("meldbase storage: document already exists")

ErrDocumentExists distinguishes an insert collision from an optimistic update/delete precondition conflict. The public DB maps it to ErrDuplicateID so a grouped candidate can fall back to ordinary per-request semantics.

View Source 
var ErrReclamationConflict = errors.New("meldbase storage: online reclamation conflicted with a commit")

Functions

func DecodeDatabaseRoot 

func DecodeDatabaseRoot(page []byte, expectedID uint64) (DatabaseRoot, Page, error)

func EncodeDatabaseRoot 

func EncodeDatabaseRoot(pageID, generation uint64, root DatabaseRoot) ([]byte, error)

func EncodeMeta 

func EncodeMeta(meta Meta) ([]byte, error)

func EncodePage 

func EncodePage(value Page) ([]byte, error)

func Open 

func Open(path string) (*File, Meta, error)

func OpenForQualification 

func OpenForQualification(path string, options OpenOptions, hook func(QualificationBoundary) error) (*File, Meta, OpenReport, error)

OpenForQualification installs a synchronous boundary hook before returning an otherwise normal file. The hook may block while an external controller cuts power, exhausts a disposable volume or terminates the process. This entry point lives in internal/storage so external applications cannot opt into qualification behavior accidentally.

func OpenWithOptions 

func OpenWithOptions(path string, options OpenOptions) (*File, Meta, OpenReport, error)

func OpenWithReport 

func OpenWithReport(path string) (*File, Meta, OpenReport, error)

Types

type BeginIndexBuildTransaction 

type BeginIndexBuildTransaction struct {
	BuildID         [16]byte
	Collection      string
	Name            string
	FieldPath       string
	Fields          []IndexField
	Unique          bool
	ReplaceExisting bool
	CreatedAt       time.Time
}

type CollectionMeta 

type CollectionMeta struct {
	ID               uint32
	PrimaryRoot      uint64
	OrderRoot        uint64
	IndexCatalogRoot uint64
	DocumentCount    uint64
	CreatedSequence  uint64
	UpdatedSequence  uint64
	// NextDocumentPosition is the greatest insertion position ever assigned in
	// this collection. Positions are never reused: update preserves one, while
	// delete followed by insert receives a newer position.
	NextDocumentPosition uint64
}

type CollectionPrecondition 

type CollectionPrecondition struct {
	Collection              string
	ExpectedExists          bool
	ExpectedID              uint32
	ExpectedUpdatedSequence uint64
}

CollectionPrecondition binds a predicate or range read to the exact collection generation observed by a transaction snapshot. It is deliberately broader than a future index-range fence: any document or published-index change to the collection invalidates it, which prevents phantoms without trusting a process-local query plan.

ExpectedID and ExpectedUpdatedSequence are required only when ExpectedExists is true. A missing collection is a valid read state and conflicts if another transaction creates it before publication.

type CollectionRecord 

type CollectionRecord struct {
	Name string
	Meta CollectionMeta
}

type CommitBatch 

type CommitBatch struct {
	Sequence      uint64
	TransactionID [16]byte
	CommittedAt   time.Time
	// CatalogRoot is the immutable post-commit catalog snapshot. Retaining it
	// in the Commit Log makes an exact historical Snapshot N reconstructible.
	CatalogRoot uint64
	Changes     []CommitChange
}

type CommitChange 

type CommitChange struct {
	CollectionID uint32
	// CollectionName is present only for named catalog changes. Ordinary
	// document events use the compact stable CollectionID exclusively.
	CollectionName string
	DocumentID     [16]byte
	Operation      CommitOperation
	ChangedPaths   []string
	Before         []byte
	After          []byte
	BeforeRef      *DocumentVersionRef
	AfterRef       *DocumentVersionRef
}

type CommitCursor 

type CommitCursor struct {
	// contains filtered or unexported fields
}

CommitCursor pins one immutable Commit Log root and replays a finite, gap-checked sequence range. A later live-stream layer tails new roots after this snapshot cursor reaches Through().

func (*CommitCursor) Close 

func (cursor *CommitCursor) Close() error

func (*CommitCursor) Next 

func (cursor *CommitCursor) Next() (CommitBatch, bool, error)

func (*CommitCursor) ResolveChange 

func (cursor *CommitCursor) ResolveChange(change CommitChange) (ResolvedCommitChange, error)

ResolveChange materializes a change returned by this finite cursor. The cursor's immutable root pin keeps referenced document versions valid until Close, even if logical retention advances concurrently.

func (*CommitCursor) Through 

func (cursor *CommitCursor) Through() uint64

type CommitOperation 

type CommitOperation uint8
const (
	CommitInsert CommitOperation = 1 + iota
	CommitUpdate
	CommitDelete
	CommitCatalog
)
const (
	DocumentInsert CommitOperation = CommitInsert
	DocumentUpdate CommitOperation = CommitUpdate
	DocumentDelete CommitOperation = CommitDelete
)

type CreateCollectionTransaction 

type CreateCollectionTransaction struct {
	TransactionID [16]byte
	CommittedAt   time.Time
	Collection    string
}

CreateCollectionTransaction creates durable empty collection metadata without manufacturing a document mutation. It is primarily the schema primitive used by logical migration, but it also gives collection creation an explicit commit identity and replay event.

type CreateIndexTransaction 

type CreateIndexTransaction struct {
	TransactionID   [16]byte
	CommittedAt     time.Time
	Collection      string
	Name            string
	FieldPath       string
	Fields          []IndexField
	Unique          bool
	ReplaceExisting bool
	// Entries is consumed and may be reordered/populated by ApplyCreateIndex.
	// Callers must not reuse or mutate it once the call begins.
	Entries []IndexEntry
}

type DatabaseRoot 

type DatabaseRoot struct {
	CommitSequence         uint64
	CatalogRoot            uint64
	CommitLogRoot          uint64
	FreeSpaceRoot          uint64
	OldestRetainedSequence uint64
	CatalogGeneration      uint64
	DocumentCount          uint64
	CollectionCount        uint64
	IndexBuildCatalogRoot  uint64
}

type DocumentIterator 

type DocumentIterator struct {
	// contains filtered or unexported fields
}

DocumentIterator streams one collection from a pinned immutable snapshot. It owns an independent reader pin, so the ReadSnapshot used to create it may be closed immediately. DocumentIterator is not safe for concurrent use.

func (*DocumentIterator) Close 

func (iterator *DocumentIterator) Close() error

func (*DocumentIterator) Err 

func (iterator *DocumentIterator) Err() error

func (*DocumentIterator) Next 

func (iterator *DocumentIterator) Next() bool

func (*DocumentIterator) Record 

func (iterator *DocumentIterator) Record() DocumentRecord

type DocumentMutation 

type DocumentMutation struct {
	Collection   string
	DocumentID   [16]byte
	Operation    CommitOperation
	Document     []byte
	ChangedPaths []string
	// Indexes must contain exactly one entry for every index currently defined
	// on the collection. BeforeKey/AfterKey are encoded scalar keys; an empty key
	// means the indexed field is absent.
	Indexes []IndexMutation
}

type DocumentPrecondition 

type DocumentPrecondition struct {
	Collection     string
	DocumentID     [16]byte
	ExpectedExists bool
	ExpectedHash   [32]byte
}

DocumentPrecondition binds an optimistic transaction to the canonical document bytes it actually read. ExpectedHash is SHA-256 over the decoded storage document body when ExpectedExists is true and must otherwise be zero.

type DocumentRecord 

type DocumentRecord struct {
	DocumentID        [16]byte
	InsertionPosition uint64
	Document          []byte
}

type DocumentSystemTransaction 

type DocumentSystemTransaction struct {
	DocumentTransaction DocumentTransaction
	SystemRecords       []SystemRecordMutation
}

DocumentSystemTransaction atomically publishes document mutations and a bounded set of private system-record changes. Every CAS must match or no document/system mutation is published.

type DocumentTransaction 

type DocumentTransaction struct {
	TransactionID           [16]byte
	CommittedAt             time.Time
	Preconditions           []DocumentPrecondition
	CollectionPreconditions []CollectionPrecondition
	Mutations               []DocumentMutation
}

type DocumentVersionRef 

type DocumentVersionRef struct {
	PrimaryRoot uint64
	DocumentID  [16]byte
}

type DurableCommitConsumer 

type DurableCommitConsumer struct {
	// contains filtered or unexported fields
}

DurableCommitConsumer is a pull-based, crash-resumable Commit Log consumer. Closing releases only its process-local replay pin; its named checkpoint remains durable and continues to cap retention until DeleteDurableCommitConsumer is called deliberately.

func (*DurableCommitConsumer) Ack 

func (consumer *DurableCommitConsumer) Ack(sequence uint64) error

Ack durably advances this consumer through a delivered sequence. Ack is monotonic and idempotent; a stale process may acknowledge an older sequence but can never move the durable checkpoint backward. The control-plane update has no logical commit sequence, so it cannot create an acknowledgement loop.

func (*DurableCommitConsumer) Checkpoint 

func (consumer *DurableCommitConsumer) Checkpoint() uint64

func (*DurableCommitConsumer) Close 

func (consumer *DurableCommitConsumer) Close() error

func (*DurableCommitConsumer) Next 

func (consumer *DurableCommitConsumer) Next(ctx context.Context) (CommitBatch, error)

Next returns the next durable Commit Log batch. It does not advance the checkpoint; callers must Ack only after their own side effect is complete.

func (*DurableCommitConsumer) ResolveChange 

func (consumer *DurableCommitConsumer) ResolveChange(change CommitChange) (ResolvedCommitChange, error)

ResolveChange materializes one change from the batch most recently returned by Next. It has the same lifetime/ordering rule as LiveCommitStream: callers must resolve all required images before advancing the consumer again.

type FailIndexBuildTransaction 

type FailIndexBuildTransaction struct {
	BuildID   [16]byte
	Failure   IndexBuildFailure
	UpdatedAt time.Time
}

type File 

type File struct {
	// contains filtered or unexported fields
}

func (*File) AbortIndexBuild 

func (f *File) AbortIndexBuild(buildID [16]byte) error

AbortIndexBuild removes only the protected build record. Its private pages become reclaimable under the normal dual-Meta epoch rules.

func (*File) ApplyCreateCollection 

func (f *File) ApplyCreateCollection(transaction CreateCollectionTransaction) (uint64, error)

ApplyCreateCollection atomically publishes an empty Primary tree, an empty IndexCatalog tree, CollectionMeta, and one matching catalog CommitBatch.

func (*File) ApplyCreateIndex 

func (f *File) ApplyCreateIndex(transaction CreateIndexTransaction) (uint64, error)

ApplyCreateIndex atomically publishes an immutable secondary tree, its index catalog entry, the owning collection metadata, and one matching CommitBatch.

func (*File) ApplyDocumentSystemTransaction 

func (f *File) ApplyDocumentSystemTransaction(transaction DocumentSystemTransaction) (SystemRecordResult, error)

ApplyDocumentSystemTransaction applies all document mutations only if every system-record precondition matches. A mismatch returns Applied=false and the isolated current value of the first mismatch without publishing any change.

func (*File) ApplyDocumentTransaction 

func (f *File) ApplyDocumentTransaction(transaction DocumentTransaction) (uint64, error)

ApplyDocumentTransaction atomically publishes document roots, catalog roots, and one matching durable CommitBatch. Duplicate document mutations inside one transaction are rejected so each logical change has unambiguous before/after version roots.

func (*File) ApplyDocumentTransactionGroup 

func (f *File) ApplyDocumentTransactionGroup(transactions []DocumentTransaction) ([]uint64, error)

ApplyDocumentTransactionGroup builds several ordered logical document commits in one physical copy-on-write generation. It is an internal storage primitive for the future CommitCoordinator; callers receive one sequence per logical member, while recovery observes either the whole group or none of it. System-record CAS is intentionally excluded until its separate idempotency and rollback-anchor matrix has group evidence.

func (*File) ApplyIndexBuildCatchUpBatch 

func (f *File) ApplyIndexBuildCatchUpBatch(batch IndexBuildCatchUpBatch) (IndexBuildMeta, error)

ApplyIndexBuildCatchUpBatch applies caller-derived keys for every relevant Commit Log change in one contiguous sequence interval. The storage layer validates sequence continuity, operation/document identity and immutable Before/After document positions before mutating the private tree.

func (*File) ApplyIndexBuildScanBatch 

func (f *File) ApplyIndexBuildScanBatch(batch IndexBuildScanBatch) (IndexBuildMeta, error)

ApplyIndexBuildScanBatch adds one Primary-key-ordered, bounded batch to the private Secondary tree and advances its durable scan cursor. Source records are resolved through the build's protected CatalogRoot, never current state.

func (*File) ApplyReplicationNoop 

func (f *File) ApplyReplicationNoop(transactionID [16]byte, committedAt time.Time) (uint64, error)

ApplyReplicationNoop advances a replica's logical Commit Log position for a source batch whose public projection is empty. The marker is private to the target's own Commit Log: it carries no user document or System-record value, but preserves the contiguous token contract required before a later public batch can be applied.

Replication of private System-record contents is deliberately not implied by this method. A follower is read-only and must not serve source-side RPC or idempotency ownership; those records need a separately authenticated control plane if they are ever replicated.

func (*File) ApplySystemRecordTransaction 

func (f *File) ApplySystemRecordTransaction(transaction SystemRecordTransaction) (SystemRecordResult, error)

func (*File) BeginIndexBuild 

func (f *File) BeginIndexBuild(transaction BeginIndexBuildTransaction) (IndexBuildMeta, error)

BeginIndexBuild protects the current CatalogRoot and one empty shadow Secondary root in a physical maintenance generation. It does not advance the logical commit sequence or publish an ordinary index definition.

func (*File) Close 

func (f *File) Close() error

func (*File) CommitRoot 

func (f *File) CommitRoot(root DatabaseRoot) error

CommitRoot appends one experimental DatabaseRoot generation and publishes it through the inactive meta slot. Tree roots referenced by root must already be durable; the first vertical slice uses zero roots to exercise atomicity.

func (*File) CopyPhysicalToContext 

func (f *File) CopyPhysicalToContext(ctx context.Context, destination io.Writer) (PhysicalCopyResult, error)

CopyPhysicalToContext copies the complete page-aligned file while holding a storage read lock. Writers are blocked for the copy duration; readers remain available. The caller owns destination creation, fsync, verification and publication.

func (*File) CreateDurableCommitConsumer 

func (f *File) CreateDurableCommitConsumer(name string, after uint64) (*DurableCommitConsumer, error)

CreateDurableCommitConsumer establishes a new named checkpoint at after. The selected position must already be recoverable from the current retained Commit Log. Creation changes only private control-plane state: it publishes a physical COW generation but does not advance the logical commit sequence or inject a synthetic change into the consumer's stream.

func (*File) DatabaseRoot 

func (f *File) DatabaseRoot() (DatabaseRoot, error)

func (*File) DeleteDurableCommitConsumer 

func (f *File) DeleteDurableCommitConsumer(name string) error

DeleteDurableCommitConsumer explicitly removes a checkpoint and its durable retention pin. It does not close another process's active stream; that stream still owns its temporary pin until Close, but future acknowledgements fail.

func (*File) FailIndexBuild 

func (f *File) FailIndexBuild(transaction FailIndexBuildTransaction) (IndexBuildMeta, error)

FailIndexBuild durably stops automatic progress without publishing or deleting the private tree. Failed builds no longer pin Commit Log history; their source/shadow pages remain reachable until explicit abort.

func (*File) FinalizeIndexBuild 

func (f *File) FinalizeIndexBuild(transaction FinalizeIndexBuildTransaction) (uint64, error)

FinalizeIndexBuild atomically moves an already-current shadow root into the ordinary IndexCatalog, removes its build record and appends one catalog Commit Log event. The Secondary tree is neither copied nor rebuilt.

func (*File) GetDocument 

func (f *File) GetDocument(collection string, documentID [16]byte) ([]byte, bool, error)

func (*File) GetSystemRecord 

func (f *File) GetSystemRecord(key []byte) ([]byte, bool, error)

func (*File) IndexBuild 

func (f *File) IndexBuild(buildID [16]byte) (IndexBuildMeta, bool, error)

IndexBuild returns one current build record without pinning it after return.

func (*File) IndexBuilds 

func (f *File) IndexBuilds() ([]IndexBuildMeta, error)

func (*File) Meta 

func (f *File) Meta() Meta

func (*File) OpenCommitCursor 

func (f *File) OpenCommitCursor(after uint64) (*CommitCursor, error)

func (*File) OpenDurableCommitConsumer 

func (f *File) OpenDurableCommitConsumer(name string) (*DurableCommitConsumer, error)

OpenDurableCommitConsumer resumes a named checkpoint. It fails explicitly if the checkpoint predates retained history; a caller must resynchronize rather than silently starting from a newer position.

func (*File) OpenIndexBuildCatchUpSnapshot 

func (f *File) OpenIndexBuildCatchUpSnapshot(buildID [16]byte) (IndexBuildMeta, *ReadSnapshot, error)

OpenIndexBuildCatchUpSnapshot atomically reads the current build record and pins the current DatabaseRoot. Its Commit Log and every document-version root referenced by retained commits cannot be reclaimed until the snapshot closes.

func (*File) OpenIndexBuildScanIterator 

func (f *File) OpenIndexBuildScanIterator(buildID [16]byte, limit int) (IndexBuildMeta, *DocumentIterator, error)

OpenIndexBuildScanIterator streams the next protected source-Primary batch. Its reader pin protects the current DatabaseRoot, which in turn protects the build record, source CatalogRoot and shadow tree for the iterator lifetime.

func (*File) OpenLiveCommitStream 

func (f *File) OpenLiveCommitStream(after uint64) (*LiveCommitStream, error)

func (*File) OpenSnapshot 

func (f *File) OpenSnapshot() (*ReadSnapshot, error)

OpenSnapshot pins only the current DatabaseRoot. It is the query/read path; callers that also need an N+1 stream use OpenSnapshotAndStream instead.

func (*File) OpenSnapshotAndStream 

func (f *File) OpenSnapshotAndStream() (*ReadSnapshot, *LiveCommitStream, error)

OpenSnapshotAndStream atomically pins Snapshot N and creates a stream whose first possible result is N+1, eliminating the query/watch registration gap.

func (*File) OpenSnapshotAndStreamAt 

func (f *File) OpenSnapshotAndStreamAt(sequence uint64) (*ReadSnapshot, *LiveCommitStream, error)

OpenSnapshotAndStreamAt atomically reconstructs historical Snapshot N and opens a durable stream whose first result is N+1. The snapshot is available only while N remains in the retained Commit Log window. Sequence zero denotes the known empty state before commit 1.

func (*File) PageCacheStats 

func (f *File) PageCacheStats() PageCacheStats

func (*File) PersistFreeSpace 

func (f *File) PersistFreeSpace() error

PersistFreeSpace publishes the current audited pool as an optional physical maintenance generation without advancing the logical commit sequence.

func (*File) PersistFreeSpaceContext 

func (f *File) PersistFreeSpaceContext(ctx context.Context) error

func (*File) Reachability 

func (f *File) Reachability() (ReachabilityStats, error)

Reachability audits all pages protected by both fallback metas and active snapshot pins. It is intentionally explicit and potentially expensive; the commit hot path never runs it.

func (*File) ReachabilityContext 

func (f *File) ReachabilityContext(ctx context.Context) (ReachabilityStats, error)

func (*File) ReadCommit 

func (f *File) ReadCommit(rootPage, sequence uint64) (CommitBatch, error)

func (*File) ReadDocumentVersion 

func (f *File) ReadDocumentVersion(reference DocumentVersionRef) ([]byte, error)

func (*File) ReclaimPages 

func (f *File) ReclaimPages() (ReachabilityStats, error)

ReclaimPages performs an explicit protected-root audit and makes every page unreachable from both valid Meta roots and all active readers available to subsequent copy-on-write transactions. It does not mutate disk by itself.

func (*File) ReclaimPagesContext 

func (f *File) ReclaimPagesContext(ctx context.Context) (ReachabilityStats, error)

func (*File) ReclaimPagesOptimisticContext 

func (f *File) ReclaimPagesOptimisticContext(ctx context.Context, maxAttempts int, persistFreeSpace bool) (ReachabilityStats, int, error)

ReclaimPagesOptimisticContext scans a duplicate read handle without holding the writer mutex. Installation takes the mutex only long enough to prove that Meta generation, physical high-water mark and the previous free pool are unchanged. A concurrent commit causes a bounded retry rather than extending the commit pause across the full graph walk.

func (*File) RetainCommitsFrom 

func (f *File) RetainCommitsFrom(keepFrom uint64, transactionID [16]byte, committedAt time.Time) (uint64, error)

RetainCommitsFrom atomically drops logical Commit Log entries older than keepFrom and appends an auditable retention commit. Physical page reclamation remains a separate epoch-safe phase.

func (*File) StorageStats 

func (f *File) StorageStats() StorageStats

StorageStats returns a bounded snapshot intended for low-frequency admin sampling. It does not traverse database trees or perform file IO.

func (*File) TreeGet 

func (f *File) TreeGet(rootPage uint64, kind TreeKind, key []byte) ([]byte, bool, error)

func (*File) TreeScan 

func (f *File) TreeScan(rootPage uint64, kind TreeKind, start, end []byte, limit int) ([]KeyValue, error)

func (*File) Update 

func (f *File) Update(build func(*WriteTxn) (DatabaseRoot, error)) error

Update builds and atomically publishes one copy-on-write generation. The callback must not retain tx and must not perform network or user callbacks.

func (*File) ValidateCollectionPreconditions 

func (f *File) ValidateCollectionPreconditions(preconditions []CollectionPrecondition) error

ValidateCollectionPreconditions checks broad collection snapshot fences without publishing a generation. ApplyDocumentTransaction performs the same validation inside its atomic write transaction, which is the required commit path for phantom-safe callers.

func (*File) ValidateDocumentPreconditions 

func (f *File) ValidateDocumentPreconditions(preconditions []DocumentPrecondition) error

ValidateDocumentPreconditions checks one point read set against the current immutable root under the file read lock without publishing a generation.

type FinalizeIndexBuildTransaction 

type FinalizeIndexBuildTransaction struct {
	BuildID                 [16]byte
	TransactionID           [16]byte
	ExpectedAppliedSequence uint64
	CommittedAt             time.Time
}

type IndexAuditFunc 

type IndexAuditFunc func(IndexMeta, [16]byte, []byte) (key []byte, indexed bool, err error)

IndexAuditFunc recomputes one logical Secondary key from the canonical stored document. It is used only by the offline verifier; normal open, reachability, reclamation and commit paths never invoke it.

type IndexBuildCatchUpBatch 

type IndexBuildCatchUpBatch struct {
	BuildID                 [16]byte
	ExpectedAppliedSequence uint64
	ThroughSequence         uint64
	Mutations               []IndexBuildCatchUpMutation
	UpdatedAt               time.Time
}

type IndexBuildCatchUpMutation 

type IndexBuildCatchUpMutation struct {
	Sequence   uint64
	DocumentID [16]byte
	Operation  CommitOperation
	BeforeKey  []byte
	AfterKey   []byte
}

type IndexBuildFailure 

type IndexBuildFailure uint8
const (
	IndexBuildFailureNone IndexBuildFailure = iota
	IndexBuildFailureUniqueConflict
	IndexBuildFailureResourceLimit
	IndexBuildFailureHistoryLost
	IndexBuildFailureCanceled
	IndexBuildFailureInvalidIndex
)

type IndexBuildMeta 

type IndexBuildMeta struct {
	BuildID           [16]byte
	CollectionID      uint32
	Collection        string
	Name              string
	FieldPath         string
	Fields            []IndexField
	Unique            bool
	ReplaceExisting   bool
	Phase             IndexBuildPhase
	Failure           IndexBuildFailure
	SourceSequence    uint64
	SourceCatalogRoot uint64
	// AppliedCatalogRoot is the immutable catalog snapshot represented by the
	// shadow after catch-up advances beyond SourceSequence. Zero is the legacy
	// encoding and is equivalent to SourceCatalogRoot only while both sequences
	// are equal.
	AppliedCatalogRoot uint64
	ShadowRoot         uint64
	ScanAfter          [16]byte
	AppliedSequence    uint64
	EntryCount         uint64
	CanonicalBytes     uint64
	CreatedAt          time.Time
	UpdatedAt          time.Time
}

type IndexBuildPhase 

type IndexBuildPhase uint8
const (
	IndexBuildScan IndexBuildPhase = 1 + iota
	IndexBuildCatchUp
	IndexBuildReady
	IndexBuildFailed
)

type IndexBuildScanBatch 

type IndexBuildScanBatch struct {
	BuildID           [16]byte
	ExpectedScanAfter [16]byte
	ScanAfter         [16]byte
	Entries           []IndexEntry
	Complete          bool
	UpdatedAt         time.Time
}

type IndexEntry 

type IndexEntry struct {
	Key               []byte
	InsertionPosition uint64
	DocumentID        [16]byte
}

type IndexField 

type IndexField struct {
	Path      string
	Direction int8
}

IndexField is one ordered component of a Secondary key. FieldPath on the surrounding metadata remains a compatibility mirror of Fields[0].Path.

type IndexIterator 

type IndexIterator struct {
	// contains filtered or unexported fields
}

IndexIterator streams one Secondary tree from a pinned immutable snapshot. It owns an independent reader pin and is not safe for concurrent use.

func (*IndexIterator) Close 

func (iterator *IndexIterator) Close() error

func (*IndexIterator) Entry 

func (iterator *IndexIterator) Entry() IndexEntry

func (*IndexIterator) Err 

func (iterator *IndexIterator) Err() error

func (*IndexIterator) Next 

func (iterator *IndexIterator) Next() bool

type IndexMeta 

type IndexMeta struct {
	Name            string
	FieldPath       string
	Fields          []IndexField
	Unique          bool
	Root            uint64
	EntryCount      uint64
	CreatedSequence uint64
	UpdatedSequence uint64
	KeyCodecVersion uint16
}

func DecodeIndexMeta 

func DecodeIndexMeta(name string, encoded []byte) (IndexMeta, error)

DecodeIndexMeta decodes an immutable index catalog value carried in a Commit Log catalog event. It is intentionally a narrow export: callers get semantic index metadata, never an invitation to depend on the physical catalog layout.

type IndexMutation 

type IndexMutation struct {
	Name      string
	BeforeKey []byte
	AfterKey  []byte
}

type KeyValue 

type KeyValue struct {
	Key, Value []byte
}

type LiveCommitStream 

type LiveCommitStream struct {
	// contains filtered or unexported fields
}

LiveCommitStream tails durable commits after a sequence. It does not keep a per-subscriber event queue: waiters wake, read the current root, and replay their own next sequence from the durable Commit Log.

func (*LiveCommitStream) Close 

func (stream *LiveCommitStream) Close() error

func (*LiveCommitStream) Next 

func (stream *LiveCommitStream) Next(ctx context.Context) (CommitBatch, error)

func (*LiveCommitStream) ResolveChange 

func (stream *LiveCommitStream) ResolveChange(change CommitChange) (ResolvedCommitChange, error)

ResolveChange materializes a change from the most recently delivered live batch. Calling Next again acknowledges that batch and may advance the replay retention lease, so callers must resolve all required images before Next.

type Meta 

type Meta struct {
	DatabaseID             [16]byte
	Generation             uint64
	CommitSequence         uint64
	RootPage               uint64
	PhysicalPageCount      uint64
	OldestRetainedSequence uint64
	RequiredFeatures       uint64
	OptionalFeatures       uint64
}

func DecodeMeta 

func DecodeMeta(page []byte) (Meta, error)

type MetaEnvelope 

type MetaEnvelope struct {
	Revision          uint16
	HeaderSize        uint16
	PageSize          uint32
	DatabaseID        [16]byte
	Generation        uint64
	CommitSequence    uint64
	RootPage          uint64
	PhysicalPageCount uint64
	RequiredFeatures  uint64
	OptionalFeatures  uint64
}

MetaEnvelope contains only fields whose byte positions are frozen for format negotiation. It can be decoded from a checksum-valid future revision without interpreting that revision's page graph.

func InspectMetaEnvelope 

func InspectMetaEnvelope(page []byte) (MetaEnvelope, error)

InspectMetaEnvelope validates only the stable magic/full-page checksum envelope and returns negotiation fields even when Revision is newer than this binary. Callers must not treat it as validation of the referenced page graph.

type MutableTree 

type MutableTree struct {
	// contains filtered or unexported fields
}

func (*MutableTree) Delete 

func (tree *MutableTree) Delete(key []byte) (bool, error)

Delete removes a key using copy-on-write path mutation. Empty children are eliminated and adjacent siblings are merged whenever the combined node fits; it never rebuilds the complete tree.

func (*MutableTree) Flush 

func (tree *MutableTree) Flush() (uint64, error)

func (*MutableTree) Get 

func (tree *MutableTree) Get(key []byte) ([]byte, bool, error)

func (*MutableTree) Put 

func (tree *MutableTree) Put(key, value []byte) error

func (*MutableTree) Scan 

func (tree *MutableTree) Scan(start, end []byte, limit int) ([]KeyValue, error)

Scan returns keys in bytewise order in [start, end). A nil bound is open and a non-positive limit is unbounded.

type OpenOptions 

type OpenOptions struct {
	RequireClean bool
	// RequireGraphAudit verifies every page protected by either valid Meta root
	// before Open succeeds. It is deliberately opt-in: ordinary  startup is
	// metadata-only so large databases do not pay an unbounded open pause.
	// Unlike VerifyPathContext it is a structural audit only; callers needing
	// Secondary-to-document semantic proof must run the offline verifier.
	RequireGraphAudit bool
	// RequirePrivateFileMode rejects a database whose existing Unix permission
	// bits grant group or other access. It never chmods an operator-owned file:
	// callers choose this fail-closed boundary explicitly for deployments where
	// the database can contain secrets.
	RequirePrivateFileMode    bool
	ExpectedDatabaseID        [16]byte
	MinimumCommitSequence     uint64
	MinimumGeneration         uint64
	CommitRetentionMaxCommits uint64
	CommitRetentionMaxBytes   uint64
	MaxFileBytes              uint64
}

type OpenReport 

type OpenReport struct {
	Created                bool
	SelectedMetaSlot       uint8
	ChecksumValidMetaSlots uint8
	RootValidMetaSlots     uint8
	MetaRedundancyDegraded bool
	FallbackToOlderRoot    bool
	TrailingBytesRemoved   uint64
	FreeSpaceLoadDegraded  bool
}

OpenReport describes only recovery decisions made while opening a file. It contains no path or user data and is immutable after OpenWithReport returns.

type Page 

type Page struct {
	Type         PageType
	Flags        uint8
	ID           uint64
	Generation   uint64
	BornSequence uint64
	ItemCount    uint32
	Link         uint64
	Payload      []byte
}

func DecodePage 

func DecodePage(page []byte, expectedID uint64) (Page, error)

type PageCacheStats 

type PageCacheStats struct {
	CapacityPages uint64
	ResidentPages uint64
	Hits          uint64
	Misses        uint64
	Evictions     uint64
}

type PageType 

type PageType uint8
const (
	PageDatabaseRoot PageType = 1 + iota
	PageCatalogBranch
	PageCatalogLeaf
	PagePrimaryBranch
	PagePrimaryLeaf
	PageSecondaryBranch
	PageSecondaryLeaf
	PageDocumentOverflow
	PageCommitLogBranch
	PageCommitLogLeaf
	PageCommitOverflow
	PageFreeSpaceBranch
	PageFreeSpaceLeaf
	PageIndexCatalogBranch
	PageIndexCatalogLeaf
	PageOrderBranch
	PageOrderLeaf
	PageSystemBranch
	PageSystemLeaf
	PageSystemOverflow
	PageIndexBuildCatalogBranch
	PageIndexBuildCatalogLeaf
)

type PhysicalCopyResult 

type PhysicalCopyResult struct {
	Meta   Meta
	Bytes  uint64
	SHA256 [32]byte
}

type QualificationBoundary 

type QualificationBoundary string

QualificationBoundary identifies a physical publication boundary exposed only to repository-internal destructive qualification runners. It is not a storage or application API contract. 

const (
	QualificationAfterPageWrite QualificationBoundary = "after-page-write"
	QualificationBeforeDataSync QualificationBoundary = "before-data-sync"
	QualificationAfterDataSync  QualificationBoundary = "after-data-sync"
	QualificationAfterMetaWrite QualificationBoundary = "after-meta-write"
	QualificationAfterMetaSync  QualificationBoundary = "after-meta-sync"
)

type ReachabilityStats 

type ReachabilityStats struct {
	PhysicalPages    uint64
	ReachablePages   uint64
	ReclaimablePages uint64
	PinnedSnapshots  uint64
}

type ReadSnapshot 

type ReadSnapshot struct {
	// contains filtered or unexported fields
}

ReadSnapshot pins one immutable DatabaseRoot. Reclamation must retain every page reachable through the snapshot until Close.

func (*ReadSnapshot) Close 

func (snapshot *ReadSnapshot) Close() error

func (*ReadSnapshot) CollectionMeta 

func (snapshot *ReadSnapshot) CollectionMeta(collection string) (CollectionMeta, bool, error)

CollectionMeta returns metadata from this snapshot's immutable CatalogRoot, including the stable numeric collection ID used by Commit Log changes.

func (*ReadSnapshot) Collections 

func (snapshot *ReadSnapshot) Collections() ([]CollectionRecord, error)

func (*ReadSnapshot) GetDocument 

func (snapshot *ReadSnapshot) GetDocument(collection string, documentID [16]byte) ([]byte, bool, error)

func (*ReadSnapshot) GetDocumentRecord 

func (snapshot *ReadSnapshot) GetDocumentRecord(collection string, documentID [16]byte) (DocumentRecord, bool, error)

GetDocumentRecord resolves one primary record including its stable insertion position, which query execution uses as the deterministic sort tie breaker.

func (*ReadSnapshot) IndexMeta 

func (snapshot *ReadSnapshot) IndexMeta(collection, name string) (IndexMeta, bool, error)

func (*ReadSnapshot) Indexes 

func (snapshot *ReadSnapshot) Indexes(collection string) ([]IndexMeta, error)

func (*ReadSnapshot) OpenCollectionIterator 

func (snapshot *ReadSnapshot) OpenCollectionIterator(collection string, start, end *[16]byte, limit int) (*DocumentIterator, error)

OpenCollectionIterator creates a bounded-memory scan over [start, end). A nil bound is open and a non-positive limit is unbounded.

func (*ReadSnapshot) OpenIndexIterator 

func (snapshot *ReadSnapshot) OpenIndexIterator(collection, name string, start, end []byte, limit int) (*IndexIterator, error)

OpenIndexIterator creates a bounded-memory Secondary scan over encoded scalar keys in [start, end). The iterator owns a snapshot pin independently.

func (*ReadSnapshot) OpenInsertionOrderIterator 

func (snapshot *ReadSnapshot) OpenInsertionOrderIterator(collection string, start, end *uint64, limit int) (*DocumentIterator, error)

OpenInsertionOrderIterator streams documents by their durable insertion position over [start, end). It resolves every Order entry through the Primary tree from the same snapshot and owns an independent reader pin.

func (*ReadSnapshot) ReadCommit 

func (snapshot *ReadSnapshot) ReadCommit(sequence uint64) (CommitBatch, error)

ReadCommit resolves one commit from this snapshot's pinned CommitLogRoot. DocumentVersionRef values returned by it remain protected until Close.

func (*ReadSnapshot) ReadDocumentVersion 

func (snapshot *ReadSnapshot) ReadDocumentVersion(reference DocumentVersionRef) ([]byte, error)

ReadDocumentVersion reads a version reference obtained from a commit in this same pinned snapshot. The reader pin prevents reclamation/reuse of its Primary root while the version is decoded.

func (*ReadSnapshot) ScanCollection 

func (snapshot *ReadSnapshot) ScanCollection(collection string, start, end *[16]byte, limit int) ([]DocumentRecord, error)

func (*ReadSnapshot) ScanIndex 

func (snapshot *ReadSnapshot) ScanIndex(collection, name string, start, end []byte, limit int) ([]IndexEntry, error)

ScanIndex returns entries whose encoded scalar key is in [start, end). Nil bounds are open. DocumentID remains an implicit deterministic key suffix.

func (*ReadSnapshot) ScanSystemRecords 

func (snapshot *ReadSnapshot) ScanSystemRecords(start, end []byte, limit int) ([]KeyValue, error)

ScanSystemRecords returns private records in bytewise [start,end) order. A non-positive limit is unbounded. It exists for bounded first-party retention maintenance, not application queries.

func (*ReadSnapshot) Sequence 

func (snapshot *ReadSnapshot) Sequence() uint64

func (*ReadSnapshot) SystemRecords 

func (snapshot *ReadSnapshot) SystemRecords() ([]KeyValue, error)

SystemRecords returns an isolated key/value snapshot for maintenance such as compact-to-new-file. It is not exposed through the public collection API.

type ResolvedCommitChange 

type ResolvedCommitChange struct {
	CollectionID uint32
	DocumentID   [16]byte
	Operation    CommitOperation
	ChangedPaths []string
	Before       []byte
	After        []byte
}

ResolvedCommitChange owns materialized before/after images. Empty images retain their operation-specific meaning (insert has no Before, delete has no After). ChangedPaths and image bytes never alias Commit Log or page-cache data.

type SortedTreeBuilder 

type SortedTreeBuilder struct {
	// contains filtered or unexported fields
}

SortedTreeBuilder consumes strictly increasing keys and builds immutable leaves eagerly. Finish constructs only the much smaller branch frontier. This is the storage boundary used by index construction today and by a future external-merge source without changing catalog publication.

A builder belongs to one WriteTxn, is not concurrency-safe, and must not be retained after the transaction callback returns.

func (*SortedTreeBuilder) Add 

func (builder *SortedTreeBuilder) Add(key, value []byte) error

Add consumes one key/value pair. Keys must be globally strictly increasing.

func (*SortedTreeBuilder) Finish 

func (builder *SortedTreeBuilder) Finish() (uint64, error)

Finish returns the immutable root page. Calling it more than once fails.

type StorageStats 

type StorageStats struct {
	PageSize                   uint64
	Generation                 uint64
	PhysicalPages              uint64
	CommitSequence             uint64
	OldestRetainedSequence     uint64
	RetainedCommits            uint64
	CommitRetentionMax         uint64
	CommitRetentionOverage     uint64
	RetainedCommitBytes        uint64
	CommitRetentionMaxBytes    uint64
	CommitRetentionByteOverage uint64
	RetentionPrunedCommits     uint64
	RetentionPressureEvents    uint64
	RetentionPressure          bool
	StorageUsedBytes           uint64
	StorageMaxBytes            uint64
	StorageByteOverage         uint64
	StorageLimitRejections     uint64
	StorageQuotaExhausted      bool
	ActiveReaders              uint64
	ActiveReplayLeases         uint64
	DocumentCount              uint64
	CollectionCount            uint64
	ReusablePages              uint64
	TreeSplits                 uint64
	TreeMerges                 uint64
	PersistentFreeSpace        bool
	FreeSpaceLoads             uint64
	FreeSpaceLoadFailures      uint64
	FreeSpacePublishes         uint64
	FreeSpaceCandidateChecks   uint64
	PageCache                  PageCacheStats
}

StorageStats is a bounded point-in-time view of physical storage health. It contains no keys, document identifiers, paths, or user values.

type SystemRecordMutation 

type SystemRecordMutation struct {
	Key            []byte
	ExpectedExists bool
	ExpectedHash   [32]byte
	NewValue       []byte
	Delete         bool
	Unconditional  bool
}

SystemRecordMutation is one compare-and-set change in the private system tree. It is also reusable by composite storage transactions that publish business and control-plane state under one DatabaseRoot.

type SystemRecordResult 

type SystemRecordResult struct {
	Sequence uint64
	Applied  bool
	Current  []byte
}

type SystemRecordTransaction 

type SystemRecordTransaction struct {
	TransactionID  [16]byte
	CommittedAt    time.Time
	Key            []byte
	ExpectedExists bool
	ExpectedHash   [32]byte
	NewValue       []byte
	Delete         bool
	Unconditional  bool
}

SystemRecordTransaction performs one compare-and-swap in the private system tree. ExpectedHash is SHA-256 over the decoded current value when ExpectedExists is true. NewValue must be non-empty unless Delete is set.

type TreeIterator 

type TreeIterator struct {
	// contains filtered or unexported fields
}

TreeIterator walks one immutable B+Tree root without materializing the result set. Key and Value point into validated immutable cached pages and remain valid until the iterator is closed or garbage collected. TreeIterator is not safe for concurrent use.

A raw TreeIterator does not pin a DatabaseRoot. Callers that can run page reclamation must hold a snapshot pin for its lifetime. DocumentIterator does this automatically.

func (*TreeIterator) Close 

func (iterator *TreeIterator) Close() error

func (*TreeIterator) Err 

func (iterator *TreeIterator) Err() error

func (*TreeIterator) Key 

func (iterator *TreeIterator) Key() []byte

func (*TreeIterator) Next 

func (iterator *TreeIterator) Next() bool

Next advances to the next entry in bytewise key order within [start, end).

func (*TreeIterator) Value 

func (iterator *TreeIterator) Value() []byte

type TreeKind 

type TreeKind uint8
const (
	TreeCatalog TreeKind = 1 + iota
	TreePrimary
	TreeSecondary
	TreeCommitLog
	TreeIndexCatalog
	TreeOrder
	TreeSystem
	TreeFreeSpace
	TreeIndexBuildCatalog
)

type VerificationResult 

type VerificationResult struct {
	Meta                        Meta
	FileBytes                   uint64
	TrailingBytes               uint64
	PhysicalPages               uint64
	ReachablePages              uint64
	ReclaimablePages            uint64
	ValidMetaSlots              int
	PersistentFreeSpace         bool
	FreeSpaceValid              bool
	SemanticIndexesVerified     bool
	SemanticIndexBuildsVerified bool
	SHA256                      [sha256.Size]byte
}

VerificationResult describes a read-only, full protected-page graph audit. ReachablePages excludes the two Meta pages and optional FreeSpace acceleration pages, matching ReachabilityStats.

func VerifyPathContext 

func VerifyPathContext(ctx context.Context, path string) (result VerificationResult, resultErr error)

VerifyPathContext opens an existing file read-only under a non-blocking shared advisory lock. It never creates, truncates, repairs, reclaims, or publishes a Meta generation. A running writer's exclusive lock fails closed.

func VerifyPathContextWithIndexAudit 

func VerifyPathContextWithIndexAudit(ctx context.Context, path string, indexAudit IndexAuditFunc) (result VerificationResult, resultErr error)

VerifyPathContextWithIndexAudit additionally proves logical Secondary keys against canonical stored documents. The callback runs only under the offline shared lock and must be deterministic, bounded and side-effect free.

type WriteTxn 

type WriteTxn struct {
	// contains filtered or unexported fields
}

WriteTxn stages immutable pages in memory. Nothing becomes visible until the DatabaseRoot and inactive MetaPage are durably published by File.Update.

func (*WriteTxn) AppendCommit 

func (tx *WriteTxn) AppendCommit(rootPage uint64, batch CommitBatch) (uint64, error)

AppendCommit writes one complete logical batch into the Commit Log tree. It must be called inside the File.Update callback for the same sequence.

func (*WriteTxn) AppendCommitRetained 

func (tx *WriteTxn) AppendCommitRetained(rootPage, oldest uint64, batch CommitBatch) (uint64, uint64, error)

AppendCommitRetained appends a business commit and advances the logical retention watermark in the same COW publication. Active replay pins cap the watermark; they never lose history to satisfy the configured window.

func (*WriteTxn) BaseRoot 

func (tx *WriteTxn) BaseRoot() DatabaseRoot

func (*WriteTxn) NewSortedTreeBuilder 

func (tx *WriteTxn) NewSortedTreeBuilder(kind TreeKind) (*SortedTreeBuilder, error)

NewSortedTreeBuilder starts an empty immutable tree build.

func (*WriteTxn) OpenTree 

func (tx *WriteTxn) OpenTree(rootPage uint64, kind TreeKind) (*MutableTree, error)

func (*WriteTxn) Sequence 

func (tx *WriteTxn) Sequence() uint64

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.