store

package
v1.0.0-rc.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 2, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

README

Evolve Storage System

The store package provides a persistent storage solution for Evolve, designed to efficiently store and retrieve blockchain data such as blocks, signatures, state, and metadata.

Overview

The storage system consists of a key-value store interface that allows for the persistence of blockchain data. It leverages the IPFS Datastore interface (go-datastore) with a Badger database implementation by default.

Badger options are tuned for the ev-node write pattern (append-heavy with periodic overwrites) via store.BadgerOptions(). Use tools/db-bench to validate performance against Badger defaults.

Core Components

Storage Interface

The main interface (Store) defines methods for:

  • Block data storage and retrieval
  • State management
  • Metadata storage
  • Height tracking and querying
Implementation

The DefaultStore is the standard implementation of the Store interface, utilizing a key-value datastore.

Data Organization

The store organizes data using a prefix-based key system:

Prefix Purpose Key Format
h Block headers /h/{height}
d Block data /d/{height}
i Block index (hash -> height) /i/{hash}
c Block signatures /c/{height}
s Chain state s
m Metadata /m/{key}

Block Storage Sequence

sequenceDiagram
    participant App as Application
    participant Store as DefaultStore
    participant DS as Datastore

    App->>Store: SaveBlockData(header, data, signature)
    Store->>Store: Prepare batch
    Store->>DS: Put header
    Store->>DS: Put data
    Store->>DS: Put signature
    Store->>DS: Put block hash → height index
    Store->>DS: Commit batch

    App->>Store: GetBlockData(height)
    Store->>DS: Get header
    DS->>Store: Return header blob
    Store->>Store: Unmarshal header
    Store->>DS: Get data
    DS->>Store: Return data blob
    Store->>Store: Unmarshal data
    Store->>App: Return header, data

Store Component Architecture

classDiagram
    class Store {
        <<interface>>
        +Height() uint64
        +SetHeight(ctx, height)
        +SaveBlockData(ctx, header, data, signature) error
        +GetBlockData(ctx, height) (header, data, error)
        +GetBlockByHash(ctx, hash) (header, data, error)
        +GetSignature(ctx, height) (signature, error)
        +GetSignatureByHash(ctx, hash) (signature, error)
        +UpdateState(ctx, state) error
        +GetState(ctx) (state, error)
        +SetMetadata(ctx, key, value) error
        +GetMetadata(ctx, key) (value, error)
        +Close() error
    }

    class DefaultStore {
        -db ds.Batching
        -height atomic.Uint64
        +Height() uint64
        +SetHeight(ctx, height)
        +SaveBlockData(ctx, header, data, signature) error
        +GetBlockData(ctx, height) (header, data, error)
        +GetBlockByHash(ctx, hash) (header, data, error)
        +GetSignature(ctx, height) (signature, error)
        +GetSignatureByHash(ctx, hash) (signature, error)
        +UpdateState(ctx, state) error
        +GetState(ctx) (state, error)
        +SetMetadata(ctx, key, value) error
        +GetMetadata(ctx, key) (value, error)
        +Close() error
    }

    class Datastore {
        <<interface>>
        +Put(key, value) error
        +Get(key) (value, error)
        +Has(key) (exists, error)
        +Delete(key) error
        +Query(q query.Query) (Results, error)
        +Close() error
    }

    Store <|.. DefaultStore
    DefaultStore --> Datastore : uses

Usage Examples

Creating a Store
// In-memory store (for testing)
kvStore, err := store.NewTestInMemoryKVStore()
if err != nil {
    // handle error
}
myStore := store.New(kvStore)

// Persistent store
kvStore, err := store.NewDefaultKVStore("/path/to/root", "data", "ev-db")
if err != nil {
    // handle error
}
myStore := store.New(kvStore)
Saving and Retrieving Data
// Save block data
err := myStore.SaveBlockData(ctx, header, data, signature)

// Get block by height
header, data, err := myStore.GetBlockData(ctx, height)

// Get block by hash
header, data, err := myStore.GetBlockByHash(ctx, blockHash)

// Update state
err := myStore.UpdateState(ctx, newState)

// Get current state
state, err := myStore.GetState(ctx)

// Store metadata
err := myStore.SetMetadata(ctx, "myKey", []byte("myValue"))

// Retrieve metadata
value, err := myStore.GetMetadata(ctx, "myKey")

Advanced Usage: Batching Operations

For performance-critical operations, the underlying datastore supports batching:

batch, err := kvStore.Batch(ctx)
if err != nil {
    // handle error
}

// Add operations to batch
batch.Put(ctx, key1, value1)
batch.Put(ctx, key2, value2)
batch.Delete(ctx, key3)

// Commit all operations atomically
err = batch.Commit(ctx)

Store Adapters for P2P Integration

The store package provides adapter implementations that wrap the ev-node store to satisfy the header.Store[H] interface from the go-header library. This enables the ev-node store to be used directly by go-header's P2P infrastructure, eliminating data duplication.

Documentation

Index

Constants

View Source 
const (
	// DefaultHeaderCacheSize is the default number of headers to cache in memory.
	DefaultHeaderCacheSize = 200_000

	// DefaultBlockDataCacheSize is the default number of block data entries to cache.
	DefaultBlockDataCacheSize = 200_000
)
View Source 
const (
	// GenesisDAHeightKey is the key used for persisting the first DA included height in store.
	// It avoids to walk over the HeightToDAHeightKey to find the first DA included height.
	GenesisDAHeightKey = "gdh"

	// HeightToDAHeightKey is the key prefix used for persisting the mapping from a Evolve height
	// to the DA height where the block's header/data was included.
	// Full keys are like: rhb/<evolve_height>/h and rhb/<evolve_height>/d
	HeightToDAHeightKey = "rhb"

	// DAIncludedHeightKey is the key used for persisting the da included height in store.
	DAIncludedHeightKey = "d"

	// LastBatchDataKey is the key used for persisting the last batch data in store.
	LastBatchDataKey = "l"

	// LastSubmittedHeaderHeightKey is the key used for persisting the last submitted header height in store.
	LastSubmittedHeaderHeightKey = "last-submitted-header-height"
)
View Source 
const EvPrefix = "0"

EvPrefix is used in KV store to separate ev-node data from execution environment data (if the same data base is reused)

Variables

This section is empty.

Functions

func BadgerOptions 

func BadgerOptions() *badger4.Options

BadgerOptions returns ev-node tuned Badger options for the node workload. These defaults favor write throughput for append-heavy usage.

func GenerateKey 

func GenerateKey(fields []string) string

GenerateKey creates a key from a slice of string fields, joining them with slashes.

func GetHeightToDAHeightDataKey 

func GetHeightToDAHeightDataKey(height uint64) string

GetHeightToDAHeightDataKey returns the metadata key for storing the DA height where a block's data was included for a given sequencer height.

func GetHeightToDAHeightHeaderKey 

func GetHeightToDAHeightHeaderKey(height uint64) string

GetHeightToDAHeightHeaderKey returns the metadata key for storing the DA height where a block's header was included for a given sequencer height.

func GetPrefixEntries 

func GetPrefixEntries(ctx context.Context, store ds.Datastore, prefix string) (dsq.Results, error)

GetPrefixEntries retrieves all entries in the datastore whose keys have the supplied prefix

func NewDefaultKVStore 

func NewDefaultKVStore(rootDir, dbPath, dbName string) (ds.Batching, error)

NewDefaultKVStore creates instance of default key-value store.

func NewEvNodeKVStore 

func NewEvNodeKVStore(kvStore ds.Batching) ds.Batching

NewEvNodeKVStore creates a new key-value store with EvPrefix prefix applied to all keys.

func NewPrefixKVStore 

func NewPrefixKVStore(kvStore ds.Batching, prefix string) ds.Batching

NewPrefixKVStore creates a new key-value store with a prefix applied to all keys.

func NewTestInMemoryKVStore 

func NewTestInMemoryKVStore() (ds.Batching, error)

NewTestInMemoryKVStore builds KVStore that works in-memory (without accessing disk).

Types

type Batch 

type Batch interface {
	// SaveBlockData atomically saves the block header, data, and signature
	SaveBlockData(header *types.SignedHeader, data *types.Data, signature *types.Signature) error

	// SetHeight sets the height in the batch
	SetHeight(height uint64) error

	// UpdateState updates the state in the batch
	UpdateState(state types.State) error

	// Commit commits all batch operations atomically
	Commit() error

	// Put adds a put operation to the batch (used internally for rollback)
	Put(key ds.Key, value []byte) error

	// Delete adds a delete operation to the batch (used internally for rollback)
	Delete(key ds.Key) error
}

Batch provides atomic operations for the store

type CachedStore 

type CachedStore struct {
	Store
	// contains filtered or unexported fields
}

CachedStore wraps a Store with LRU caching for frequently accessed data. The underlying LRU cache is thread-safe, so no additional synchronization is needed.

func NewCachedStore 

func NewCachedStore(store Store, opts ...CachedStoreOption) (*CachedStore, error)

NewCachedStore creates a new CachedStore wrapping the given store.

func (*CachedStore) ClearCache 

func (cs *CachedStore) ClearCache()

ClearCache clears all cached entries.

func (*CachedStore) Close 

func (cs *CachedStore) Close() error

Close closes the underlying store.

func (*CachedStore) GetBlockData 

func (cs *CachedStore) GetBlockData(ctx context.Context, height uint64) (*types.SignedHeader, *types.Data, error)

GetBlockData returns block header and data at given height, using cache if available.

func (*CachedStore) GetHeader 

func (cs *CachedStore) GetHeader(ctx context.Context, height uint64) (*types.SignedHeader, error)

GetHeader returns the header at the given height, using the cache if available.

func (*CachedStore) InvalidateRange 

func (cs *CachedStore) InvalidateRange(fromHeight, toHeight uint64)

InvalidateRange removes headers in the given range from the cache.

func (*CachedStore) Rollback 

func (cs *CachedStore) Rollback(ctx context.Context, height uint64, aggregator bool) error

Rollback wraps the underlying store's Rollback and invalidates affected cache entries.

type CachedStoreOption 

type CachedStoreOption func(*CachedStore) error

CachedStoreOption configures a CachedStore.

func WithBlockDataCacheSize 

func WithBlockDataCacheSize(size int) CachedStoreOption

WithBlockDataCacheSize sets the block data cache size.

func WithHeaderCacheSize 

func WithHeaderCacheSize(size int) CachedStoreOption

WithHeaderCacheSize sets the header cache size.

type DataStoreAdapter 

type DataStoreAdapter = StoreAdapter[*types.Data]

func NewDataStoreAdapter 

func NewDataStoreAdapter(store Store, gen genesis.Genesis) *DataStoreAdapter

NewDataStoreAdapter creates a new StoreAdapter for data. The genesis is used to determine the initial height for efficient Tail lookups.

type DataStoreGetter 

type DataStoreGetter struct {
	// contains filtered or unexported fields
}

DataStoreGetter implements StoreGetter for *types.Data.

func NewDataStoreGetter 

func NewDataStoreGetter(store Store) *DataStoreGetter

NewDataStoreGetter creates a new DataStoreGetter.

func (*DataStoreGetter) GetByHash 

func (g *DataStoreGetter) GetByHash(ctx context.Context, hash []byte) (*types.Data, error)

GetByHash implements StoreGetter.

func (*DataStoreGetter) GetByHeight 

func (g *DataStoreGetter) GetByHeight(ctx context.Context, height uint64) (*types.Data, error)

GetByHeight implements StoreGetter.

func (*DataStoreGetter) HasAt 

func (g *DataStoreGetter) HasAt(ctx context.Context, height uint64) bool

HasAt implements StoreGetter.

func (*DataStoreGetter) Height 

func (g *DataStoreGetter) Height(ctx context.Context) (uint64, error)

Height implements StoreGetter.

type DefaultBatch 

type DefaultBatch struct {
	// contains filtered or unexported fields
}

DefaultBatch provides a batched write interface for atomic store operations

func (*DefaultBatch) Commit 

func (b *DefaultBatch) Commit() error

Commit commits all batched operations atomically

func (*DefaultBatch) Delete 

func (b *DefaultBatch) Delete(key ds.Key) error

Delete adds a delete operation to the batch

func (*DefaultBatch) Put 

func (b *DefaultBatch) Put(key ds.Key, value []byte) error

Put adds a put operation to the batch

func (*DefaultBatch) SaveBlockData 

func (b *DefaultBatch) SaveBlockData(header *types.SignedHeader, data *types.Data, signature *types.Signature) error

SaveBlockData saves block data to the batch

func (*DefaultBatch) SetHeight 

func (b *DefaultBatch) SetHeight(height uint64) error

SetHeight sets the height in the batch

func (*DefaultBatch) UpdateState 

func (b *DefaultBatch) UpdateState(state types.State) error

UpdateState updates the state in the batch

type DefaultStore 

type DefaultStore struct {
	// contains filtered or unexported fields
}

DefaultStore is a default store implementation.

func (*DefaultStore) Close 

func (s *DefaultStore) Close() error

Close safely closes underlying data storage, to ensure that data is actually saved.

func (*DefaultStore) GetBlockByHash 

func (s *DefaultStore) GetBlockByHash(ctx context.Context, hash []byte) (*types.SignedHeader, *types.Data, error)

GetBlockByHash returns block with given block header hash, or error if it's not found in Store.

func (*DefaultStore) GetBlockData 

func (s *DefaultStore) GetBlockData(ctx context.Context, height uint64) (*types.SignedHeader, *types.Data, error)

GetBlockData returns block header and data at given height, or error if it's not found in Store.

func (*DefaultStore) GetHeader 

func (s *DefaultStore) GetHeader(ctx context.Context, height uint64) (*types.SignedHeader, error)

GetHeader returns the header at the given height or error if it's not found in Store.

func (*DefaultStore) GetMetadata 

func (s *DefaultStore) GetMetadata(ctx context.Context, key string) ([]byte, error)

GetMetadata returns values stored for given key with SetMetadata.

func (*DefaultStore) GetSignature 

func (s *DefaultStore) GetSignature(ctx context.Context, height uint64) (*types.Signature, error)

GetSignature returns signature for a block with given block header hash, or error if it's not found in Store.

func (*DefaultStore) GetSignatureByHash 

func (s *DefaultStore) GetSignatureByHash(ctx context.Context, hash []byte) (*types.Signature, error)

GetSignatureByHash returns signature for a block at given height, or error if it's not found in Store.

func (*DefaultStore) GetState 

func (s *DefaultStore) GetState(ctx context.Context) (types.State, error)

GetState returns last state saved with UpdateState.

func (*DefaultStore) GetStateAtHeight 

func (s *DefaultStore) GetStateAtHeight(ctx context.Context, height uint64) (types.State, error)

GetStateAtHeight returns the state at the given height. If no state is stored at that height, it returns an error.

func (*DefaultStore) Height 

func (s *DefaultStore) Height(ctx context.Context) (uint64, error)

Height returns height of the highest block saved in the Store.

func (*DefaultStore) NewBatch 

func (s *DefaultStore) NewBatch(ctx context.Context) (Batch, error)

NewBatch creates a new batch for atomic operations

func (*DefaultStore) Rollback 

func (s *DefaultStore) Rollback(ctx context.Context, height uint64, aggregator bool) error

Rollback rolls back block data until the given height from the store. When aggregator is true, it will check the latest data included height and prevent rollback further than that. NOTE: this function does not rollback metadata. Those should be handled separately if required. Other stores are not rolled back either.

func (*DefaultStore) SetMetadata 

func (s *DefaultStore) SetMetadata(ctx context.Context, key string, value []byte) error

SetMetadata saves arbitrary value in the store.

Metadata is separated from other data by using prefix in KV.

func (*DefaultStore) Sync 

func (s *DefaultStore) Sync(ctx context.Context) (err error)

Sync flushes the store state to disk. Returns nil if the database has been closed (common during shutdown).

type HeaderStoreAdapter 

type HeaderStoreAdapter = StoreAdapter[*types.SignedHeader]

Type aliases for convenience

func NewHeaderStoreAdapter 

func NewHeaderStoreAdapter(store Store, gen genesis.Genesis) *HeaderStoreAdapter

NewHeaderStoreAdapter creates a new StoreAdapter for headers. The genesis is used to determine the initial height for efficient Tail lookups.

type HeaderStoreGetter 

type HeaderStoreGetter struct {
	// contains filtered or unexported fields
}

HeaderStoreGetter implements StoreGetter for *types.SignedHeader.

func NewHeaderStoreGetter 

func NewHeaderStoreGetter(store Store) *HeaderStoreGetter

NewHeaderStoreGetter creates a new HeaderStoreGetter.

func (*HeaderStoreGetter) GetByHash 

func (g *HeaderStoreGetter) GetByHash(ctx context.Context, hash []byte) (*types.SignedHeader, error)

GetByHash implements StoreGetter.

func (*HeaderStoreGetter) GetByHeight 

func (g *HeaderStoreGetter) GetByHeight(ctx context.Context, height uint64) (*types.SignedHeader, error)

GetByHeight implements StoreGetter.

func (*HeaderStoreGetter) HasAt 

func (g *HeaderStoreGetter) HasAt(ctx context.Context, height uint64) bool

HasAt implements StoreGetter.

func (*HeaderStoreGetter) Height 

func (g *HeaderStoreGetter) Height(ctx context.Context) (uint64, error)

Height implements StoreGetter.

type Reader 

type Reader interface {
	// Height returns height of the highest block in store.
	Height(ctx context.Context) (uint64, error)

	// GetBlockData returns block at given height, or error if it's not found in Store.
	GetBlockData(ctx context.Context, height uint64) (*types.SignedHeader, *types.Data, error)
	// GetBlockByHash returns block with given block header hash, or error if it's not found in Store.
	GetBlockByHash(ctx context.Context, hash []byte) (*types.SignedHeader, *types.Data, error)
	// GetSignature returns signature for a block at given height, or error if it's not found in Store.
	GetSignature(ctx context.Context, height uint64) (*types.Signature, error)
	// GetSignatureByHash returns signature for a block with given block header hash, or error if it's not found in Store.
	GetSignatureByHash(ctx context.Context, hash []byte) (*types.Signature, error)
	// GetHeader returns the header at given height, or error if it's not found in Store.
	GetHeader(ctx context.Context, height uint64) (*types.SignedHeader, error)

	// GetState returns last state saved with UpdateState.
	GetState(ctx context.Context) (types.State, error)
	// GetStateAtHeight returns state saved at given height, or error if it's not found in Store.
	GetStateAtHeight(ctx context.Context, height uint64) (types.State, error)

	// GetMetadata returns values stored for given key with SetMetadata.
	GetMetadata(ctx context.Context, key string) ([]byte, error)
}

type Rollback 

type Rollback interface {
	// Rollback deletes x height from the ev-node store.
	// Aggregator is used to determine if the rollback is performed on the aggregator node.
	Rollback(ctx context.Context, height uint64, aggregator bool) error
}

type Store 

type Store interface {
	Rollback
	Reader

	// SetMetadata saves arbitrary value in the store.
	//
	// This method enables evolve to safely persist any information.
	SetMetadata(ctx context.Context, key string, value []byte) error

	// Close safely closes underlying data storage, to ensure that data is actually saved.
	Close() error

	// NewBatch creates a new batch for atomic operations.
	NewBatch(ctx context.Context) (Batch, error)
}

Store is minimal interface for storing and retrieving blocks, commits and state.

func New 

func New(ds ds.Batching) Store

New returns new, default store.

func WithTracingStore 

func WithTracingStore(inner Store) Store

WithTracingStore wraps a Store with OpenTelemetry tracing.

type StoreAdapter 

type StoreAdapter[H header.Header[H]] struct {
	// contains filtered or unexported fields
}

StoreAdapter is a generic adapter that wraps Store to implement header.Store[H]. This allows the ev-node store to be used directly by go-header's P2P infrastructure, eliminating the need for a separate go-header store and reducing data duplication.

The adapter maintains an in-memory cache for items received via P2P (through Append). This cache allows the go-header syncer and P2P handler to access items before they are validated and persisted by the ev-node syncer. Once the ev-node syncer processes a block, it writes to the underlying store, and subsequent reads will come from the store.

func NewStoreAdapter 

func NewStoreAdapter[H header.Header[H]](getter StoreGetter[H], gen genesis.Genesis) *StoreAdapter[H]

NewStoreAdapter creates a new StoreAdapter wrapping the given store getter. The genesis is used to determine the initial height for efficient Tail lookups.

func (*StoreAdapter[H]) Append 

func (a *StoreAdapter[H]) Append(ctx context.Context, items ...H) error

Append stores items in the pending cache. These items are received via P2P and will be available for retrieval until the ev-node syncer processes and persists them to the store.

func (*StoreAdapter[H]) DeleteRange 

func (a *StoreAdapter[H]) DeleteRange(ctx context.Context, from, to uint64) error

DeleteRange deletes items in the range [from, to). This is used for rollback operations.

func (*StoreAdapter[H]) Get 

func (a *StoreAdapter[H]) Get(ctx context.Context, hash header.Hash) (H, error)

Get returns an item by its hash.

func (*StoreAdapter[H]) GetByHeight 

func (a *StoreAdapter[H]) GetByHeight(ctx context.Context, height uint64) (H, error)

GetByHeight returns an item at the given height. If the height is not yet available, it blocks until it is or context is canceled.

func (*StoreAdapter[H]) GetRange 

func (a *StoreAdapter[H]) GetRange(ctx context.Context, from, to uint64) ([]H, error)

GetRange returns items in the range [from, to).

func (*StoreAdapter[H]) GetRangeByHeight 

func (a *StoreAdapter[H]) GetRangeByHeight(ctx context.Context, from H, to uint64) ([]H, error)

GetRangeByHeight returns items in the range [from.Height()+1, to). This follows go-header's convention where 'from' is the trusted item and we return items starting from the next height.

func (*StoreAdapter[H]) Has 

func (a *StoreAdapter[H]) Has(ctx context.Context, hash header.Hash) (bool, error)

Has checks if an item with the given hash exists.

func (*StoreAdapter[H]) HasAt 

func (a *StoreAdapter[H]) HasAt(ctx context.Context, height uint64) bool

HasAt checks if an item exists at the given height.

func (*StoreAdapter[H]) Head 

func (a *StoreAdapter[H]) Head(ctx context.Context, _ ...header.HeadOption[H]) (H, error)

Head returns the highest item in the store.

func (*StoreAdapter[H]) Height 

func (a *StoreAdapter[H]) Height() uint64

Height returns the current height of the store.

func (*StoreAdapter[H]) Init 

func (a *StoreAdapter[H]) Init(ctx context.Context, item H) error

Init initializes the store with the first item. This is called by go-header when bootstrapping the store with a trusted item.

func (*StoreAdapter[H]) OnDelete 

func (a *StoreAdapter[H]) OnDelete(fn func(context.Context, uint64) error)

OnDelete registers a callback to be invoked when items are deleted.

func (*StoreAdapter[H]) Start 

func (a *StoreAdapter[H]) Start(ctx context.Context) error

Start implements header.Store. It initializes the adapter if needed.

func (*StoreAdapter[H]) Stop 

func (a *StoreAdapter[H]) Stop(ctx context.Context) error

Stop implements header.Store. No-op since the underlying store lifecycle is managed separately.

func (*StoreAdapter[H]) Sync 

func (a *StoreAdapter[H]) Sync(ctx context.Context) error

Sync ensures all pending writes are flushed. No-op for the adapter as pending data is in-memory cache.

func (*StoreAdapter[H]) Tail 

func (a *StoreAdapter[H]) Tail(ctx context.Context) (H, error)

Tail returns the lowest item in the store. For ev-node, this is typically the genesis/initial height. If pruning has occurred, it walks up from initialHeight to find the first available item. TODO(@julienrbrt): Optimize this when pruning is enabled.

type StoreGetter 

type StoreGetter[H header.Header[H]] interface {
	// GetByHeight retrieves an item by its height.
	GetByHeight(ctx context.Context, height uint64) (H, error)
	// GetByHash retrieves an item by its hash.
	GetByHash(ctx context.Context, hash []byte) (H, error)
	// Height returns the current height of the store.
	Height(ctx context.Context) (uint64, error)
	// HasAt checks if an item exists at the given height.
	HasAt(ctx context.Context, height uint64) bool
}

StoreGetter abstracts the store access methods for different types (headers vs data).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.