gormbulk

README

gorm-bulk

Perform regular Gorm tasks - in bulk!

This project aims to support the missing feature from the famous ORM Gorm. The feature I'm talking about is bulk support which has been on the wish list since 2014; see here.

This is inspired by t-tiger/gorm-bulk-insert which in turn is inspired by this comment but with the focus on flexibility and letting the end user handle final SQL, final values, how many to bulk at once etcetera.

Project status

This project is in it's early phase and since I want to ensure that the end user interface ends up as smoot, simple and flexible as possible I won't create a v1.0 release tag until I feel the most important things are in to place.

This doesn't mean that things aren't workign as indented, just that the API might change without notice.

Installation

go get -u github.com/bombsimon/gorm-bulk/...

Usage

Generate slice conversion

To be able to iterate over any type you have to pass an interface slice ([]interface{}). To make this easier this package is bundled with a code generator that will generate functions to convert []*<T> and []<T> to []interface{}.

See exmaples for details about how to use go generate and what the result will look like.

Bulk actions

This package ships with a few standard bulk action methods. A bulk action uses an ExecFunc which takes a *gorm.Scope (holding the table name, all the values and where you may set the SQL), a slice of all column names and a slice of all placeholder groups (a set of prepared statements for each slice element).

• InsertFunc - Regular INSERT INTO with all passed values.
• InsertIgnoreFunc - Run INSERT IGNORE INTO with all passed values which will just discard duplicates (and any other error).
• InsertOnDuplicateKeyUpdateFunc - Run INSERT INTO ... VALUES(...) ON DUPLICATE KEY UPDATE x = VALUES(x).

Notice that InsertFunc and InsertIgnoreFunc will look at gorm:insert_option to fetch any user defined additions.

These three ExecFuncs are wrapped in BulkInsert, BulkInsertIgnore and BulkInsertOnDuplicateKeyUpdate so you only have to pass your *gorm.DB and interface slice.

func Example(db *gorm.DB, myTypes []MyType) error {
    myTypesAsInterface := MyTypeSliceToInterfaceSlice(myTypes)

    if err := gormbulk.BulkInsert(db, myTypesAsInterface); err != nil {
        return err
    }

    return nil
}

Creating your own action

To create your own action where you may return whatever SQL and values you want just implement an ExecFunc. This is how a simple INSERT INTO would be defined.

func MyCustomBulkFunc(scope *gorm.Scope, columnNames, placeholders []string) {
    scope.Raw(fmt.Sprintf(
        "INSERT INTO %s (%s) VALUES %s",
        scope.QuotedTableName(),
        strings.Join(columnNames, ", "), 
        strings.Join(placeholders, ", "), 
    ))
}

Using the bulk

If you just want to perform a simple bulk insert, use one of the pre implemented wrapper bulk functions and pass your *gorm.DB and data set, see the example.

Documentation

Index

• func BulkExec(db *gorm.DB, objects []interface{}, execFunc ExecFunc) error
• func BulkExecChunk(db *gorm.DB, objects []interface{}, execFunc ExecFunc, chunkSize int) []error
• func BulkInsert(db *gorm.DB, objects []interface{}) error
• func BulkInsertIgnore(db *gorm.DB, objects []interface{}) error
• func BulkInsertOnDuplicateKeyUpdate(db *gorm.DB, objects []interface{}) error
• func InsertFunc(scope *gorm.Scope, columnNames, groups []string)
• func InsertIgnoreFunc(scope *gorm.Scope, columnNames, groups []string)
• func InsertOnDuplicateKeyUpdateFunc(scope *gorm.Scope, columnNames, groups []string)
• func ObjectToMap(object interface{}) (map[string]*gorm.Field, error)
• type ExecFunc

Functions

func BulkExec

func BulkExec(db *gorm.DB, objects []interface{}, execFunc ExecFunc) error

BulkExec will convert a slice of interface to bulk SQL statement. The final SQL will be determined by the ExecFunc passed.

func BulkExecChunk

func BulkExecChunk(db *gorm.DB, objects []interface{}, execFunc ExecFunc, chunkSize int) []error

BulkExecChunk will split the objects passed into the passed chunk size. A slice of errors will be returned (if any).

func BulkInsert

func BulkInsert(db *gorm.DB, objects []interface{}) error

BulkInsert will call BulkExec with the default InsertFunc.

func BulkInsertIgnore

func BulkInsertIgnore(db *gorm.DB, objects []interface{}) error

BulkInsertIgnore will call BulkExec with the default InsertFunc.

func BulkInsertOnDuplicateKeyUpdate

func BulkInsertOnDuplicateKeyUpdate(db *gorm.DB, objects []interface{}) error

BulkInsertOnDuplicateKeyUpdate will call BulkExec with the default InsertFunc.

func InsertFunc

func InsertFunc(scope *gorm.Scope, columnNames, groups []string)

InsertFunc is the default insert func. It will pass a gorm.Scope pointer which holds all the vars in scope.SQLVars. The value set to scope.SQL will be used as SQL and the variables in scope.SQLVars will be used as values.

INSERT INTO `tbl`
  (col1, col2)
VALUES
  (?, ?), (?, ?)

func InsertIgnoreFunc

func InsertIgnoreFunc(scope *gorm.Scope, columnNames, groups []string)

InsertIgnoreFunc will run INSERT IGNORE with all the records and values set on the passed scope pointer.

INSERT IGNORE INTO `tbl`
  (col1, col2)
VALUES
  (?, ?), (?, ?)

func InsertOnDuplicateKeyUpdateFunc

func InsertOnDuplicateKeyUpdateFunc(scope *gorm.Scope, columnNames, groups []string)

InsertOnDuplicateKeyUpdateFunc will perform a bulk insert but on duplicate key perform an update.

INSERT INTO `tbl`
  (col1, col2)
VALUES
  (?, ?), (?, ?)
ON DUPLICATE KEY UPDATE
  col1 = VALUES(col1),
  col2 = VALUES(col2)

func ObjectToMap

func ObjectToMap(object interface{}) (map[string]*gorm.Field, error)

ObjectToMap takes any object of type <T> and returns a map with the gorm field DB name as key and the value as value. Special fields and actions

• Foreign keys - Will be left out
• Relationship fields - Will be left out
• Fields marked to be ignored - Will be left out
• Fields named ID with auto increment - Will be left out
• Fields named ID set as primary key with blank value - Will be left out
• Blank fields with default value - Will be set to the default value

Types

type ExecFunc

type ExecFunc func(scope *gorm.Scope, columnNames, groups []string)