csm

README

Continuous-schema-migration

This is a golang library for on-the-fly/lazy migrations.

When doing migrations, it can be difficult to execute stop-the-world migrations and migrate all the data at once. This is a library that shall enable the migration when the data is retrieved/stored, and even a second time to convert the remaining data:

flowchart TB

Initial[Before migration]--Deploy code-->Progressive[Migrate entity when created/updated];
Progressive-->Progressive
Progressive--If completion needed-->Full[Migrate remaining data];
Full-->Migrated;

Let's have an example with a migration for a Person entity from v1 to v2:

// From

PersonV1{
    FirstName string
}

// To

PersonV2{
    FirstName string
    LastName  string
}

If we take a look at a data which is updated from the database, show as Migrate entity when created/updated in the previous diagram:

flowchart TB

OldData["{'firstName':'John','__schema_version':1}"]--Migration to last version-->Code
Code["PersonV2{\nFirstName string\nLastName string\n}"]--Save with actual version-->NewData["{'firstName':'John','lastName:'','__schema_version':2}"]

Supported formats

• BSON
• JSON

Install the library

go get github.com/lerenn/continuous-schema-migration

How-To

This how-to is based on the JSON version.

Setting up the migrator

// Create a new migrator
mig := csm.NewMigratorJSON[EntityV3](migrations)

Importing (and migrating) data from JSON

Here, the data is written but could come from a database.

// Importing an old object, do some modifications and save it as last version
data := []byte(`{"FullName":"John Robert Reddington","__schema_version":1}`)
migratedEntity, _ := mig.Import(data)

Exporting the data to JSON with version

// Exporting the object with the version field
data, _ = mig.Export(migratedEntity)

Now the data can be saved in the database.

Examples

• BSON
• JSON

Documentation

Index

• Constants
• Variables
• func WrapperBSON[A, B any](doc bson.D, fn func(A) (B, error)) (bson.D, error)
• func WrapperJSON[A, B any](data []byte, fn func(A) (B, error)) ([]byte, error)
• type MigrationBSON
• type MigrationJSON
• type Migrator
• type MigratorBSON
  • func NewMigratorBSON[T any](migrations []MigrationBSON) *MigratorBSON[T]
  • func (mj *MigratorBSON[T]) Export(entity T) (bson.D, error)
  • func (mj *MigratorBSON[T]) Import(data bson.D) (T, error)
  • func (mj *MigratorBSON[T]) LastVersion() int
• type MigratorJSON
  • func NewMigratorJSON[T any](migrations []MigrationJSON) *MigratorJSON[T]
  • func (mj *MigratorJSON[T]) Export(entity T) ([]byte, error)
  • func (mj *MigratorJSON[T]) Import(data []byte) (T, error)
  • func (mj *MigratorJSON[T]) LastVersion() int

Constants

const (
	// VersionFieldKey is the key that is used to store the schema version in the
	// objects.
	VersionFieldKey = "__schema_version"
)

Variables

var (
	// ErrGeneric is the generic error that can be used to isolate
	// continuous-schema-migration errors.
	ErrGeneric = errors.New("continuous schema migration error")
	// ErrNoVersion happens when no version is detected.
	ErrNoVersion = fmt.Errorf("%w: no version detected", ErrGeneric)
	// ErrInvalidVersionFormat happens when the version is not an integer.
	ErrInvalidVersionFormat = fmt.Errorf("%w: version is not an expected format", ErrGeneric)
	// ErrVersionNotFound happens when the given version is not found in migrations.
	ErrVersionNotFound = fmt.Errorf("%w: version not found in migrations", ErrGeneric)
	// ErrRunningMigration happens when a migration fails.
	ErrRunningMigration = fmt.Errorf("%w: running migration failed", ErrGeneric)
)

Functions

func WrapperBSON

func WrapperBSON[A, B any](doc bson.D, fn func(A) (B, error)) (bson.D, error)

WrapperBSON is function that will unwrap a BSON object into a type A, run a callback that should transform the type A object into type B, and wrap the resulting object of type B into a BSON.

func WrapperJSON

func WrapperJSON[A, B any](data []byte, fn func(A) (B, error)) ([]byte, error)

WrapperJSON is function that will unwrap a JSON object into a type A, run a callback that should transform the type A object into type B, and wrap the resulting object of type B into a JSON.

Types

type MigrationBSON

type MigrationBSON func(bson.D) (bson.D, error)

MigrationBSON is the function signature for BSON object migration.

type MigrationJSON

type MigrationJSON func([]byte) ([]byte, error)

MigrationJSON is the function signature for JSON object migration.

type Migrator

type Migrator[E, D any] interface {
	// Import imports a object from D (data) to E (entity) type object,
	// following migrations set in the migrator.
	Import(data D) (E, error)

	// Export exports a E (entity) type object into a D (data) object,
	// adding the version number to the resulting data.
	Export(entity E) (D, error)

	// LastVersion returns the last version of the migrations
	LastVersion() int
}

Migrator is the interface that every migrator should satisfy to respect libraries engagements. Note: E is the entity type and D is the Data type.

type MigratorBSON

type MigratorBSON[T any] struct {
	// contains filtered or unexported fields
}

MigratorBSON is the structure that will contains the migration and apply them on BSON to object T type object, and revert.

func NewMigratorBSON

func NewMigratorBSON[T any](migrations []MigrationBSON) *MigratorBSON[T]

NewMigratorBSON creates a new BSON migrator with given type and migrations.

func (*MigratorBSON[T]) Export

func (mj *MigratorBSON[T]) Export(entity T) (bson.D, error)

Export exports a T type object into a BSON object, adding the version number to the BSON object.

func (*MigratorBSON[T]) Import

func (mj *MigratorBSON[T]) Import(data bson.D) (T, error)

Import imports a BSON object into the T type object, following migrations set in the migrator.

func (*MigratorBSON[T]) LastVersion

func (mj *MigratorBSON[T]) LastVersion() int

LastVersion returns the last version of the migrations.

type MigratorJSON

type MigratorJSON[T any] struct {
	// contains filtered or unexported fields
}

MigratorJSON is the structure that will contains the migration and apply them on JSON to object T type object, and revert.

func NewMigratorJSON

func NewMigratorJSON[T any](migrations []MigrationJSON) *MigratorJSON[T]

NewMigratorJSON creates a new JSON migrator with given type and migrations.

func (*MigratorJSON[T]) Export

func (mj *MigratorJSON[T]) Export(entity T) ([]byte, error)

Export exports a T type object into a JSON object, adding the version number to the JSON object.

func (*MigratorJSON[T]) Import

func (mj *MigratorJSON[T]) Import(data []byte) (T, error)

Import imports a JSON object into the T type object, following migrations set in the migrator.

func (*MigratorJSON[T]) LastVersion

func (mj *MigratorJSON[T]) LastVersion() int

LastVersion returns the last version of the migrations.