migration

Documentation

Index

• Variables
• func CopyFromBadgerToPebble(badgerDB *badger.DB, pebbleDB *pebble.DB, cfg MigrationConfig) error
• func GeneratePrefixes(n int) [][]byte
• func RunMigration(badgerDir string, pebbleDir string, cfg MigrationConfig) error
• type KVPair
• type MigrationConfig

Variables

var DefaultMigrationConfig = MigrationConfig{
	BatchByteSize:          32_000_000,
	ReaderWorkerCount:      2,
	WriterWorkerCount:      2,
	ReaderShardPrefixBytes: 2,
}

Functions

func CopyFromBadgerToPebble

func CopyFromBadgerToPebble(badgerDB *badger.DB, pebbleDB *pebble.DB, cfg MigrationConfig) error

CopyFromBadgerToPebble migrates all key-value pairs from a BadgerDB instance to a PebbleDB instance.

The migration is performed in parallel using a configurable number of reader and writer workers. Reader workers iterate over the BadgerDB by sharded key prefixes (based on ReaderShardPrefixBytes) and send key-value pairs to a shared channel. Writer workers consume from this channel and write batched entries into PebbleDB.

Configuration is provided via MigrationConfig:

• BatchByteSize: maximum size in bytes for a single Pebble write batch.
• ReaderWorkerCount: number of concurrent workers reading from Badger.
• WriterWorkerCount: number of concurrent workers writing to Pebble.
• ReaderShardPrefixBytes: number of bytes used to shard the keyspace for parallel iteration.

The function blocks until all keys are migrated and written successfully. It returns an error if any part of the process fails.

func GeneratePrefixes

func GeneratePrefixes(n int) [][]byte

func RunMigration

func RunMigration(badgerDir string, pebbleDir string, cfg MigrationConfig) error

RunMigration performs a complete migration of key-value data from a BadgerDB directory to a PebbleDB directory and verifies the integrity of the migrated data.

It executes the following steps:

1. Validates that the Badger directory exists and is non-empty. Ensures that the Pebble directory does not already contain data.
2. Opens both databases and runs the migration using CopyFromBadgerToPebble with the given config.
3. Writes a "MIGRATION_STARTED" marker file with a timestamp in the Pebble directory.
4. After migration, performs validation by: - Generating a list of prefix shards (based on 2-byte prefixes) - Finding the min and max keys for each prefix group - Comparing the values of those keys between Badger and Pebble
5. Writes a "MIGRATION_COMPLETED" marker file with a timestamp to signal successful completion.

This function returns an error if any part of the process fails, including directory checks, database operations, or validation mismatches.

Types

type KVPair

type KVPair struct {
	Key   []byte
	Value []byte
}

type MigrationConfig

type MigrationConfig struct {
	BatchByteSize     int // the size of each batch to write to pebble
	ReaderWorkerCount int // the number of workers to read from badger
	WriterWorkerCount int // the number of workers to write to the pebble

	// number of prefix bytes used to assign iterator workload
	// e.g, if the number is 1, it means the first byte of the key is used to divide into 256 key space,
	// and each worker will be assigned to iterate all keys with the same first byte.
	// Since keys are not evenly distributed, especially some table under a certain prefix byte may have
	// a lot more data than others, we might choose to use 2 or 3 bytes to divide the key space, so that
	// the redaer worker can concurrently iterate keys with the same prefix bytes (same table).
	ReaderShardPrefixBytes int
}