package module
v0.0.0-...-86559e9 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jun 15, 2022 License: Apache-2.0 Imports: 10 Imported by: 1


dh - A simple Database Deployment Handler in Go

dh is a reaction to DBIx::Class::DeploymentHandler, a database migration system I built in in 2010. While the original DBICDH was built on top of an existing ORM and schema transformation tooling, this version is intentionally simpler, and allows users to drop in ORM support if needed.

Getting Started

The primary users of this program are expected to be Go programmers, using dh as a library. First, create your deployment directory:

mkdir -p dh/001
cat <<EOF > dh/001/dh.sql

cat <<EOF > dh/001/shortlinks.sql
CREATE TABLE shortlinks (
        PRIMARY KEY ("from")

cat <<EOF > dh/001/history.sql

Add that directory to your migration plan:

echo 000-sqlite > dh/plan.txt
echo 001       >> dh/plan.txt

Now let's create a migration to add another column:

mkdir dh/002

cat <<SQL >dh/002/shortlinks.sql
ALTER TABLE shortlinks ADD COLUMN "description" NOT NULL DEFAULT ''

cat <<SQL > dh/002/history.sql

And add it to your plan:

echo 002 >> dh/plan.txt

And here's how you'd use it:

dbh := sql.Connect(...)
m := dh.NewMigrator()
if err := m.MigrateOne(dbh, dh.DHMigrations, "000-sqlite"); err != nil {
if err := m.MigrateAll(dbh, os.DirFS("dh")); err != nil {
        return err

Out of the box dh can apply SQL or lists of SQL from JSON files, but the migration interface is extensible so you could wire up gopher-lua to do more advanced migrations.



package dh is a tool for maintaining RDBMS versions.


db := sqlx.NewDb(dbh, "sqlite3")
e := dh.NewMigrator(dh.ExtensionMigrator{})
if err := e.MigrateOne(db, dh.DHMigrations, "000-sqlite"); err != nil {
if err := e.MigrateAll(db, migrationDir); err != nil {


dh is inspired by my own DBIx::Class::DeploymentHandler, which worked well but required the use of an ORM, and additionally had some other limitations, like needing to reliably split the SQL in the migration files into statements, which ended up being pretty frustrating.

dh simplifies the situation dramatically. You create directories of migrations; out of the box these migrations can contain SQL files (defined by having the `.sql` extension) or JSON files (defined by having the `.json` extension, containing simply an array of strings to be run as SQL statements.)

In addition to directories of files, at the same level as the directories you define a plan (in `plan.txt`) that simply lists the migrations to be applied in order. Comments (starting with `#`) are ignored, as are blank lines.

$ ls -F migrations
001/  002/  plan.txt
$ cat plan.txt
000-sqlite # built into dh

## The First Migration

The first migration is special, as it must create the table that dh uses to store which migrations have been applied. To work with the built in MigrationStorage it should be named `dh_migrations` and only needs two columns, `version`, which is the directory name applied, and `id`, which must increase with each applied version, so that queries for the last `id` will return the most recently applied version.

dh includes a migration fit for the first migration that works for SQLite, which you can apply by doing something like this:

e := dh.NewMigrator(dh.ExtensionMigrator{})
err := e.MigrateOne(db, dh.DHMigrations, "000-sqlite")

If people submit MRs for other databases I'll happily have them.



This section is empty.


View Source
var DHMigrations fs.FS

DHMigrations is an fs.FS containing pre-made migrations for the migration storage. Currently contains:

  • 000-sqlite


This section is empty.


type DoesMigrate

type DoesMigrate interface {
	// Migrate applies a file to the database within the current transaction.
	Migrate(*sql.Tx, fs.File) error

DoesMigrate are things that can apply migrations to the database.

Built in examples include the JSONMigrator, SQLMigrator, and the related dispatcher, ExtensionMigrator. There's an example of a slightly more complicated migrator in the tests.

type DoesMigrationStorage

type DoesMigrationStorage interface {
	// StoreVersion inserts v in db.
	StoreVersion(db sqlx.Execer, version string) error

	// CurrentVersion returns the version stored in db.
	CurrentVersion(db sqlx.Ext) (string, error)

DoesMigrationStorage is the interface a type must conform to to be able to be used by dh.

type ExtensionMigrator

type ExtensionMigrator struct {
	// contains filtered or unexported fields

ExtensionMigrator applies migrations based on their suffix. It is intentionally simple; rather than submitting Pull Requests to add support for (eg) YAML, you are encouraged to maintain your own.

func (ExtensionMigrator) Migrate

func (m ExtensionMigrator) Migrate(dbh *sql.Tx, f fs.File) error

type JSONMigrator

type JSONMigrator struct{}

JSONMigrator applies statements in a json file read as an array of strings. For example you could have a file that looks like this:

  "INSERT INTO foo (a, b, c) VALUES (1, 2, 3)",
  "INSERT INTO foo (a, b, c) VALUES (3, 2, 1)",
  "INSERT INTO foo (a, b, c) VALUES (9, 9, 9)"

func (JSONMigrator) Migrate

func (m JSONMigrator) Migrate(dbh *sql.Tx, f fs.File) error

type MigrationStorage

type MigrationStorage struct{}

func (MigrationStorage) CurrentVersion

func (s MigrationStorage) CurrentVersion(dbh sqlx.Ext) (string, error)

func (MigrationStorage) StoreVersion

func (s MigrationStorage) StoreVersion(dbh sqlx.Execer, v string) error

type Migrator

type Migrator struct {
	// contains filtered or unexported fields

Migrator is the main type to be used within dh. It orchestrates the migrations based on the provided plan.

func NewMigrator

func NewMigrator() Migrator

NewMigrator returns a Migrator instance with MigrationStorage and the ExtensionMigrator. Create your own Migrator instace or replace those values if you need to.

func (Migrator) MigrateAll

func (m Migrator) MigrateAll(dbh *sqlx.DB, d fs.FS) error

MigrateAll applies the migrations listed in `plan.txt`, starting with the migration after the one most recently applied to the database.

func (Migrator) MigrateOne

func (m Migrator) MigrateOne(dbh *sqlx.DB, d fs.FS, version string) error

MigrateOne applies a migration directory from d named version.

Note that the version storage must exist by the time MigrateOne completes, or an error will be returned.

type Plan

type Plan struct{}

Plan parses the `plan.txt` file. It ignores comments (prefixed with `#`) and whitespace.

func (Plan) Parse

func (p Plan) Parse(r io.Reader) ([]string, error)

func (Plan) Validate

func (p Plan) Validate(ps []string, fss fs.FS) error

Validate verifies that all deployments listed in ps actually exist.

type SQLMigrator

type SQLMigrator struct{}

SQLMigrator applies a file as a single Exec call. If that will cause issues with your database, you can either switch to JSONMigrator, or add a SQL Parser to your project to properly split statements.

func (SQLMigrator) Migrate

func (m SQLMigrator) Migrate(dbh *sql.Tx, f fs.File) error

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL