dbsync

Documentation

Overview

Package dbsync contains the code for transferring data in and out of external PostgreSQL and MySQL databases, primarily for Maestro import tables.

Index

• type PrimitiveSelect
  • func (s *PrimitiveSelect) String() string
• type TableReader
  • func NewTableReader(dbConn *sqlx.DB, driver, table string, stmt *PrimitiveSelect, ...) (TableReader, error)
• type TableWriter
  • func NewTableWriter(dbConn *sqlx.DB, driver string, bqTable *bigquery.Table, table string) (TableWriter, error)

Types

type PrimitiveSelect

type PrimitiveSelect struct {
	Select  string
	From    string
	Where   string
	Limit   string
	OrderBy string
}

A primitive select is an SQL statement which only supports SELECT, FROM, WHERE, LIMIT and ORDER BY.

func (*PrimitiveSelect) String

func (s *PrimitiveSelect) String() string

Convet a PrimitiveSelect to an actual SQL statement string.

type TableReader

type TableReader interface {
	Read(p []byte) (n int, err error)

	Bytes() int64
	Rows() int64
	LastId() string
	Statement() string
	CountReport(fn func(int64, int64), every int)
	GetBQSchema(dbConn *sqlx.DB, stmt PrimitiveSelect) (*bigquery.TableSchema, error)
}

This interface should be satisfied by all the drivers. It is a Reader, as well as provides a bunch of transfer stats and the schema converted from driver-specific dialect to BigQuery.

func NewTableReader

func NewTableReader(dbConn *sqlx.DB, driver, table string, stmt *PrimitiveSelect, idColumn, lastId string, compress bool) (TableReader, error)

Create a new reader. The driver should be "potgres" or "mysql". The idColumn is the column used for incremental imports and should be indexed. lastId is the greatest id of the previous import. If compress is true, the data will be gzip compressed.

type TableWriter

type TableWriter interface {
	Begin() error
	WriteRow([]string) error
	Commit() error
}

This interface should be satisfied by all writer drivers.

func NewTableWriter

func NewTableWriter(dbConn *sqlx.DB, driver string, bqTable *bigquery.Table, table string) (TableWriter, error)

Create a new writer. Driver should be "postgres" or "mysql" (NIY). The table argument allows giving the table an alternative name in the target external database.