package module
Version: v0.0.0-...-e504d38 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Aug 8, 2021 License: MIT Imports: 9 Imported by: 0


libschema - schema migration for libraries

GoDoc unit tests pg tests


go get


Libschema provides a way for Go libraries to manage their own migrations.

Trying migrations to libraries supports two things: the first is source code locality: the migrations can be next to the code that uses the tables that the migrations address.

The second is support for migrations in third-party libraries. This is a relatively unexplored and unsolved problem: how can an open source (or proprietary) library specify and maintain a database schema. Libschema hopes to start solving this problem.

Register and execute

Migrations are registered:

schema := libschema.NewSchema(ctx, libschema.Options{})

sqlDB, err := sql.Open("postgres", "....")

database, err := lspostgres.New(logger, "main-db", schema, sqlDB)

	lspostgres.Script("createUserTable", `
			CREATE TAGLE users (
				name	text,
				id	bigint
	lspostgres.Script("addLastLogin", `
			ALTER TABLE users
				ADD COLUMN last_login timestamp

Migrations are then run run later in the order that they were registered.

err := schema.Migrate(context)

Computed Migrations

Migrations may be SQL strings or migrations can be done in Go:

	lspostgres.Computed("importUsers", func(_ context.Context, _ MyLogger, tx *sql.Tx) error {
		// code to import users here

Asynchronous migrations

The normal mode for migrations is to run the migrations synchronously when schema.Migrate() is called. Asynchronous migrations are started when schema.Migrate() is called but they're run in the background in a go-routine. If there are later migrations, after the asynchronous migration, they'll force the asynchronous migration to be synchronous unless they're also asynchronous.

Version blocking

Migrations can be tied to specific code versions so that they are not run until conditions are met. This is done with SkipRemainingIf. This be used to backfill data.

	lspostgres.Script("addColumn", `
			ALTER TABLE users
				ADD COLUMN rating`,
	libschema.SkipThisAndFollowingIf(func() bool {
		return semver.Compare(version(), "3.11.3") < 1
	lspostgres.Script("fillInRatings", `
			UPDATE	users
			SET	rating = ...
			WHERE	rating IS NULL;

			ALTER TABLE users

Cross-library dependencies

Although it is best if the schema from one library is independent of the schema for another, sometimes that's not possible, especially if you want to enforce foriegn key constraints.

Use After() to specify a cross-library dependency.

	lspostgres.Script("addOrg", `
			ALTER TABLE users
					REFERENCES org (name) `, 
	libschema.After("orgs", "createOrgTable")),

	lspostgres.Script("createOrgTable", `


For databases that support transactions on metadata, all migrations will be wrapped with a BEGIN and COMMIT. For databases that do not support transactions on metadata, migrations will be split into individual commands and run one at a time. If only some of the commands succeed, the migration will be marked as partially complete. If the migration is revised, then the later parts can be re-tried as long as the earlier parts are not modified. This does not apply to Compute()ed migrations.

Command line

Libschema adds command line flags that change the behavior of calling schema.Migrate()

--migrate-only			Call os.Exit() after completing migrations
--migrate-database		Migrate only the database named by NewDatabase
--migrate-dsn			Override *sql.DB 
--no-migrate			Skip all migrations
--exit-if-migrate-needed	Return error if migrations are not current

Ordering and pull requests

Migrations are run the order that they're defined. If the set of migrations is updated so that there are new migrations that are earlier in the table than migrations that have already run, this is not considered an error and the new migrations will be run anyway. This allows multiple branches of code with migrations to be merged into a combined branch without hassle.

Code structure

Registering the migrations before executing them suggests using library singletons. Library singletons can be supported by using nserve or fx. With nserve, migrations can be given their own hook.

Driver inclusion and database support

Like database/sql, libschema requires database-specific drivers. Include "" for Postgres support and "" for Mysql support.

libschema currently supports: PostgreSQL, MySQL. It is easy to add additional databases.

Forward only

Libschema does not support reverse migrations. If you need to fix a migration, fix forward. The history behind this is that reverse migrations are rarely the right answer for production systems and the extra work for maintaining reverse migrations is does not have enough of a payoff during development to be worth the effort.

One way to get the benefits of reverse migrations for development is to put enough enough reverse migrations to reverse to the last production schema at the end of the migration list but protected by a gateway:

libschema.SkipThisAndRemainingIf(func() bool {
	return os.Getenv("LIBMIGRATE_REVERSE_TO_PROD") != "true"

This set of reverse migrations would always be small since it would just be enough to take you back to the current production release.




View Source
const DefaultTrackingTable = "libschema.migration_status"


View Source
var EverythingSynchronous = flag.Bool("migrate-all-synchronously", false, "Run async migrations synchronously")

TreateAsyncAsRequired command line flag causes asynchronous migrations to be treated like regular migrations from the point of view of --migrate-only, --no-migrate, and --exit-if-migrate-needed.

View Source
var ExitIfMigrateNeeded = flag.Bool("exit-if-migrate-needed", false, "Return error if migrations are not current")

ExitIfMigrateNeeded command line flag causes the program to exit instead of running required migrations. Asynchronous migrations do not count as required.

View Source
var MigrateDSN = flag.String("migrate-dsn", "", "Override *sql.DB")

MigrateDSN overrides the data source name for a single database. It must be used in conjunction with MigrateDatabase.

View Source
var MigrateDatabase = flag.String("migrate-database", "", "Migrate only the database named by NewDatabase")

MigrateDatabase command line flag specifies that only a specific database should be migrated. The name corresponds to the name provided with the schema.NewDatabase() call

View Source
var MigrateOnly = flag.Bool("migrate-only", false, "Call os.Exit() after completing migrations")

MigrateOnly command line flag causes the program to exit when migrations are complete. Asynchronous migrations may be skipped.

View Source
var NoMigrate = flag.Bool("no-migrate", false, "Skip all migrations (except async)")

NoMigrate command line flag skips all migrations


func OpenAnyDB

func OpenAnyDB(dsn string) (*sql.DB, error)


type Database

type Database struct {
	Options Options
	Context context.Context
	// contains filtered or unexported fields

Database tracks all of the migrations for a specific database.

func (*Database) DB

func (d *Database) DB() *sql.DB

func (*Database) Lookup

func (d *Database) Lookup(name MigrationName) (Migration, bool)

func (*Database) Migrations

func (d *Database) Migrations(libraryName string, migrations ...Migration)

Migrations specifies the migrations needed for a library. By default, each migration is dependent upon the prior migration and they'll run in the order given. By default, all the migrations for a library will run in the order in which the library migrations are defined.

type Driver

type Driver interface {
	CreateSchemaTableIfNotExists(context.Context, MyLogger, *Database) error
	LockMigrationsTable(context.Context, MyLogger, *Database) error
	UnlockMigrationsTable(MyLogger) error

	// DoOneMigration must update the both the migration status in
	// the Database object and it must persist the migration status
	// in the tracking table.  It also does the migration.
	DoOneMigration(context.Context, MyLogger, *Database, Migration) error

	// IsMigrationSupported exists to guard against additional migration
	// options and features.  It should return nil except if there are new
	// migration features added that haven't been included in all support
	// libraries.
	IsMigrationSupported(*Database, MyLogger, Migration) error

	LoadStatus(context.Context, MyLogger, *Database) ([]MigrationName, error)

Driver interface is what's required to use libschema with a new database.

type Migration

type Migration interface {
	Base() *MigrationBase
	Copy() Migration

MigrationBase is a workaround for lacking object inheritance.

type MigrationBase

type MigrationBase struct {
	Name MigrationName
	// contains filtered or unexported fields

Migration defines a single database defintion update.

func (MigrationBase) Copy

func (m MigrationBase) Copy() MigrationBase

func (*MigrationBase) SetStatus

func (m *MigrationBase) SetStatus(status MigrationStatus)

func (*MigrationBase) Status

func (m *MigrationBase) Status() MigrationStatus

type MigrationName

type MigrationName struct {
	Name    string
	Library string

MigrationName holds both the name of the specific migration and the library to which it belongs.

func (MigrationName) String

func (n MigrationName) String() string

type MigrationOption

type MigrationOption func(Migration)

MigrationOption modifies a migration to set additional parameters

func After

func After(lib, migration string) MigrationOption

After sets up a dependency between one migration and another. This can be across library boundaries. By default, migrations are dependent on the prior migration defined. After specifies that the current migration must run after the named migration.

func Asyncrhronous

func Asyncrhronous() MigrationOption

Asyncrhronous marks a migration is okay to run asynchronously. If all of the remaining migrations can be asynchronous, then schema.Migrate() will return while the remaining migrations run.

type MigrationStatus

type MigrationStatus struct {
	Done    bool
	Partial string // for Mysql, the string represents the portion of multiple commands that have completed
	Error   string // If an attempt was made but failed, this will be set

MigrationStatus tracks if a migration is complete or not.

type MyLogger

type MyLogger interface {
	Trace(msg string, fields[string]interface{})
	Debug(msg string, fields[string]interface{})
	Info(msg string, fields[string]interface{})
	Warn(msg string, fields[string]interface{})
	Error(msg string, fields[string]interface{})

See This interface definition will not

type Options

type Options struct {
	// TrackingTable is the name of the table used to track which migrations
	// have been applied
	TrackingTable string

	// SchemaOverride is used to override the default schema.  This is most useful
	// for testing schema migrations
	SchemaOverride string

	// These TxOptions will be used for all migration transactions.
	MigrationTxOptions *sql.TxOptions

	ErrorOnUnknownMigrations bool

	// OnMigrationFailure is only called when there is a failure
	// of a specific migration.  OnMigrationsComplete will also
	// be called.
	OnMigrationFailure func(n MigrationName, err error)

	// OnMigrationsStarted is only called if migrations are needed
	OnMigrationsStarted func()

	// OnMigrationsComplete called even if no migrations are needed.  It
	// will be called when async migrations finish even if they finish
	// with an error.
	OnMigrationsComplete func(error)

	// DebugLogging turns on extra debug logging
	DebugLogging bool

Options operate at the Database level but are specified at the Schema level at least initially.

type Schema

type Schema struct {
	// contains filtered or unexported fields

Schema tracks all the migrations

func New

func New(ctx context.Context, options Options) *Schema

New creates a schema object

func (*Schema) Migrate

func (s *Schema) Migrate(ctx context.Context) (err error)

Migrate runs pending migrations that have been registered as long as the command line flags support doing migrations. We all remaining migrations are asynchronous then the remaining migrations will be run in the background.

A lock is held while migrations are in progress so that there is no chance of double migrations.

func (*Schema) NewDatabase

func (s *Schema) NewDatabase(log MyLogger, name string, db *sql.DB, driver Driver) (*Database, error)

NewDatabase creates a Database object. For Postgres, this is bundled into lspostgres.New().


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL