README

Documentation Build Status Coverage Status

goose

goose is a database migration tool.

You can manage your database's evolution by creating incremental SQL or Go scripts.

Stability notice

This repo is a fork from https://bitbucket.org/liamstask/goose. There are many things that we plan to change, mostly to make the tool easier to use, and more standardized.

As such, until this notice goes away, vendoring is highly recommended.

Install

$ go get github.com/steinbacher/goose/cmd/goose

This will install the goose binary to your $GOPATH/bin directory.

You can also build goose into your own applications by importing github.com/steinbacher/goose. Documentation is available at godoc.org.

NOTE: the API is still new, and may undergo some changes.

Omitting drivers

The default goose binary includes support for all available drivers. Sometimes this results in a lengthy build process. Drivers may be omitted from the build by using build tags.

For example

$ go get -tags nosqlite3 github.com/steinbacher/goose/cmd/goose

The available tags are:

  • nomymysql
  • nomysql
  • nopq
  • nosqlite3

Usage

goose provides several commands to help manage your database schema.

create

Create a new SQL migration.

$ goose create AddSomeColumns
$ goose: created db/migrations/20130106093224_AddSomeColumns.sql

Edit the newly created script to define the behavior of your migration.

You can also create a Go migration:

$ goose create -type go AddSomeColumns
$ goose: created db/migrations/20130106093224_AddSomeColumns.go

up

Apply all available migrations.

$ goose up
$ goose: migrating db environment 'development', current version: 0, target: 3
$ OK    001_basics.sql
$ OK    002_next.sql
$ OK    003_and_again.go
option: pgschema

Use the pgschema flag with the up command specify a postgres schema.

$ goose -pgschema=my_schema_name up
$ goose: migrating db environment 'development', current version: 0, target: 3
$ OK    001_basics.sql
$ OK    002_next.sql
$ OK    003_and_again.go

down

Roll back a single migration from the current version.

$ goose down
$ goose: migrating db environment 'development', current version: 3, target: 2
$ OK    003_and_again.go

redo

Roll back the most recently applied migration, then run it again.

$ goose redo
$ goose: migrating db environment 'development', current version: 3, target: 2
$ OK    003_and_again.go
$ goose: migrating db environment 'development', current version: 2, target: 3
$ OK    003_and_again.go

status

Print the status of all migrations:

$ goose status
$ goose: status for environment 'development'
$   Applied At                  Migration
$   =======================================
$   Sun Jan  6 11:25:03 2013 -- 001_basics.sql
$   Sun Jan  6 11:25:03 2013 -- 002_next.sql
$   Pending                  -- 003_and_again.go

dbversion

Print the current version of the database:

$ goose dbversion
$ goose: dbversion 002

goose -h provides more detailed info on each command.

Migrations

goose supports migrations written in SQL or in Go - see the goose create command above for details on how to generate them.

SQL Migrations

A sample SQL migration looks like:

-- +goose Up
CREATE TABLE post (
    id int NOT NULL,
    title text,
    body text,
    PRIMARY KEY(id)
);

-- +goose Down
DROP TABLE post;

Notice the annotations in the comments. Any statements following -- +goose Up will be executed as part of a forward migration, and any statements following -- +goose Down will be executed as part of a rollback.

By default, SQL statements are delimited by semicolons - in fact, query statements must end with a semicolon to be properly recognized by goose.

More complex statements (PL/pgSQL) that have semicolons within them must be annotated with -- +goose StatementBegin and -- +goose StatementEnd to be properly recognized. For example:

-- +goose Up
-- +goose StatementBegin
CREATE OR REPLACE FUNCTION histories_partition_creation( DATE, DATE )
returns void AS $$
DECLARE
  create_query text;
BEGIN
  FOR create_query IN SELECT
      'CREATE TABLE IF NOT EXISTS histories_'
      || TO_CHAR( d, 'YYYY_MM' )
      || ' ( CHECK( created_at >= timestamp '''
      || TO_CHAR( d, 'YYYY-MM-DD 00:00:00' )
      || ''' AND created_at < timestamp '''
      || TO_CHAR( d + INTERVAL '1 month', 'YYYY-MM-DD 00:00:00' )
      || ''' ) ) inherits ( histories );'
    FROM generate_series( $1, $2, '1 month' ) AS d
  LOOP
    EXECUTE create_query;
  END LOOP;  -- LOOP END
END;         -- FUNCTION END
$$
language plpgsql;
-- +goose StatementEnd

Go Migrations

A sample Go migration looks like:

package main

import (
    "database/sql"
    "fmt"
)

func Up_20130106222315(txn *sql.Tx) {
    fmt.Println("Hello from migration 20130106222315 Up!")
}

func Down_20130106222315(txn *sql.Tx) {
    fmt.Println("Hello from migration 20130106222315 Down!")
}

Up_20130106222315() will be executed as part of a forward migration, and Down_20130106222315() will be executed as part of a rollback.

The numeric portion of the function name (20130106222315) must be the leading portion of migration's filename, such as 20130106222315_descriptive_name.go. goose create does this by default.

A transaction is provided, rather than the DB instance directly, since goose also needs to record the schema version within the same transaction. Each migration should run as a single transaction to ensure DB integrity, so it's good practice anyway.

Configuration

goose expects you to maintain a folder (typically called "db"), which contains the following:

  • a dbconf.yaml file that describes the database configurations you'd like to use
  • a folder called "migrations" which contains .sql and/or .go scripts that implement your migrations

You may use the -path option to specify an alternate location for the folder containing your config and migrations.

A sample dbconf.yml looks like

development:
    driver: postgres
    dsn: user=liam dbname=tester sslmode=disable

Here, development specifies the name of the environment, and the driver and dsn elements are passed directly to database/sql to access the specified database.

You may include as many environments as you like, and you can use the -env command line option to specify which one to use. goose defaults to using an environment called development.

The configuration may also be environment-less, with all fields at the top level. For example:

driver: postgres
dsn: user=liam dbname=tester sslmode=disable

You can even use a mixture of both. If a field is not specified within an environment, goose will fall back to looking at the top level.

You may also include environment variables in any field of the config. Specify them as $MY_ENV_VAR or ${MY_ENV_VAR}.

Configless

Goose can also run without a config file, by pulling all parameters from environment variables. This mode operates exactly as if you passed the following config file:

migrationsDir: $DB_MIGRATIONS_DIR
driver: $DB_DRIVER
import: $DB_DRIVER_IMPORT
dialect: $DB_DIALECT
dsn: $DB_DSN

Other Drivers

goose knows about some common SQL drivers, but it can still be used to run Go-based migrations with any driver supported by database/sql. An import path and known dialect are required.

Currently, available dialects are: "postgres", "mysql", "sqlite3", and "redshift"

To run Go-based migrations with another driver, specify its import path and dialect, as shown below.

myenv:
    driver: custom
    dsn: custom open string
    import: github.com/custom/driver
    dialect: mysql

NOTE: Because migrations written in SQL are executed directly by the goose binary, only drivers compiled into goose may be used for these migrations.

Using goose with Heroku

These instructions assume that you're using Keith Rarick's Heroku Go buildpack. First, add a file to your project called (e.g.) install_goose.go to trigger building of the goose executable during deployment, with these contents:

// use build constraints to work around http://code.google.com/p/go/issues/detail?id=4210
// +build heroku

// note: need at least one blank line after build constraint
package main

import _ "github.com/steinbacher/goose/cmd/goose"

Set up your Heroku database(s) as usual.

Then make use of environment variable expansion in your dbconf.yml:

production:
    driver: postgres
    dsn: $DATABASE_URL

To run goose in production, use heroku run:

heroku run goose -env production up

Contributors

Thank you!

  • Josh Bleecher Snyder (josharian)
  • Abigail Walthall (ghthor)
  • Daniel Heath (danielrheath)
  • Chris Baynes (chris_baynes)
  • Michael Gerow (gerow)
  • Vytautas Šaltenis (rtfb)
  • James Cooper (coopernurse)
  • Gyepi Sam (gyepisam)
  • Matt Sherman (clipperhouse)
  • runner_mei
  • John Luebs (jkl1337)
  • Luke Hutton (lukehutton)
  • Kevin Gorjan (kevingorjan)
  • Brendan Fosberry (Fozz)
  • Nate Guerin (gusennan)

Documentation

Index

Constants

View Source
const (
	DirectionDown = Direction(false)
	DirectionUp   = Direction(true)
)

Variables

View Source
var (
	ErrTableDoesNotExist = errors.New("table does not exist")
	ErrNoPreviousVersion = errors.New("no previous version found")
)

Functions

func Asset

func Asset(name string) ([]byte, error)

    Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

    func AssetDir

    func AssetDir(name string) ([]string, error)

      AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy:

      data/
        foo.txt
        img/
          a.png
          b.png
      

      then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

      func AssetInfo

      func AssetInfo(name string) (os.FileInfo, error)

        AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

        func AssetNames

        func AssetNames() []string

          AssetNames returns the names of the assets.

          func CreateMigration

          func CreateMigration(name, migrationType, dir string, t time.Time) (path string, err error)

          func EnsureDBVersion

          func EnsureDBVersion(conf *DBConf, db *sql.DB) (int64, error)

            retrieve the current version for this DB. Create and initialize the DB version table if it doesn't exist.

            func FinalizeMigration

            func FinalizeMigration(conf *DBConf, txn *sql.Tx, direction Direction, v int64) error

              Update the version table for the given migration, and finalize the transaction.

              func GetDBVersion

              func GetDBVersion(conf *DBConf) (version int64, err error)

                wrapper for EnsureDBVersion for callers that don't already have their own DB instance

                func GetMostRecentDBVersion

                func GetMostRecentDBVersion(dirpath string) (version int64, err error)

                  helper to identify the most recent possible version within a folder of migration scripts

                  func GetPreviousDBVersion

                  func GetPreviousDBVersion(dirpath string, version int64) (previous int64, err error)

                  func MustAsset

                  func MustAsset(name string) []byte

                    MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

                    func NumericComponent

                    func NumericComponent(name string) (int64, error)

                      look for migration scripts with names in the form:

                      XXX_descriptivename.ext
                      

                      where XXX specifies the version number and ext specifies the type of migration

                      func OpenDBFromDBConf

                      func OpenDBFromDBConf(conf *DBConf) (*sql.DB, error)

                        OpenDBFromDBConf wraps database/sql.DB.Open() and configures the newly opened DB based on the given DBConf.

                        Callers must Close() the returned DB.

                        func RestoreAsset

                        func RestoreAsset(dir, name string) error

                          RestoreAsset restores an asset under the given directory

                          func RestoreAssets

                          func RestoreAssets(dir, name string) error

                            RestoreAssets restores an asset under the given directory recursively

                            func RunMigrations

                            func RunMigrations(conf *DBConf, migrationsDir string, target int64) (err error)

                            func RunMigrationsOnDb

                            func RunMigrationsOnDb(conf *DBConf, migrationsDir string, target int64, db *sql.DB) (err error)

                              Runs migration on a specific database instance.

                              Types

                              type DBConf

                              type DBConf struct {
                              	MigrationsDir string
                              	Driver        DBDriver
                              }

                              func NewDBConf

                              func NewDBConf(dbDir, env string) (*DBConf, error)

                                extract configuration details from the given file

                                type DBDriver

                                type DBDriver struct {
                                	Name    string
                                	DSN     string
                                	Import  string
                                	Dialect SqlDialect
                                }

                                  DBDriver encapsulates the info needed to work with a specific database driver

                                  func (*DBDriver) IsValid

                                  func (drv *DBDriver) IsValid() bool

                                    ensure we have enough info about this driver

                                    type Direction

                                    type Direction bool

                                    func (Direction) String

                                    func (d Direction) String() string

                                    type Migration

                                    type Migration struct {
                                    	Version   int64
                                    	IsApplied bool
                                    	TStamp    time.Time
                                    	Source    string // path to .go or .sql script
                                    }

                                    func CollectMigrations

                                    func CollectMigrations(dirpath string) (m []*Migration, err error)

                                      collect all the valid looking migration scripts in the migrations folder, and key them by version

                                      type MySqlDialect

                                      type MySqlDialect struct{}

                                      type PostgresDialect

                                      type PostgresDialect struct{}

                                      type RedshiftDialect

                                      type RedshiftDialect struct{}

                                      type SqlDialect

                                      type SqlDialect interface {
                                      	// contains filtered or unexported methods
                                      }

                                        SqlDialect abstracts the details of specific SQL dialects for goose's few SQL specific statements

                                        type Sqlite3Dialect

                                        type Sqlite3Dialect struct{}

                                        Directories

                                        Path Synopsis
                                        _example
                                        cmd