bqutil

README

The bqutil package

This package contains utilities that facilitate various bigquery operations, notably query processing.

Documentation

Index

• func ParseModel(row map[string]bigquery.Value, model interface{}) (interface{}, error)
• type TableInfo
• type TableUtil
  • func NewTableUtil(project, dataset string, httpClient *http.Client) (TableUtil, error)
  • func (util *TableUtil) Dedup(src string, overwrite bool, ...)
  • func (util *TableUtil) DedupInPlace(src string)
  • func (util *TableUtil) DestinationQuery(query string, dest *bigquery.Table) *bigquery.Query
  • func (util *TableUtil) GetInfoMatching(dataset, filter string) []TableInfo
  • func (util *TableUtil) GetTableStats(table string) bigquery.TableMetadata
  • func (util *TableUtil) QueryAndParse(q string, model interface{}) (interface{}, error)
  • func (util *TableUtil) ResultQuery(query string, dryRun bool) *bigquery.Query

Functions

func ParseModel

func ParseModel(row map[string]bigquery.Value, model interface{}) (interface{}, error)

ParseModel will parse a bigquery row map into a new item matching model. Type of model must be struct annotated with qfield tags.

Types

type TableInfo

type TableInfo struct {
	Name             string
	IsPartitioned    bool
	NumBytes         int64
	NumRows          uint64
	CreationTime     time.Time
	LastModifiedTime time.Time
}

TableInfo contains the critical stats for a specific table or partition.

type TableUtil

type TableUtil struct {
	Dataset *bigquery.Dataset
	// contains filtered or unexported fields
}

TableUtil provides utility functions on tables in a dataset. It encapsulates the client and Dataset to simplify methoDataset. TODO(gfr) Should this be called DatasetUtil ?

func NewTableUtil

func NewTableUtil(project, dataset string, httpClient *http.Client) (TableUtil, error)

NewTableUtil creates a TableUtil for a project. httpClient is used to inject mocks for the bigquery client. if nil, a suitable default client is used.

func (*TableUtil) Dedup

func (util *TableUtil) Dedup(src string, overwrite bool, DatasettProject, DatasettDataset, DatasettTable string)

Dedup executes a query that dedups and writes to an appropriate partition.

func (*TableUtil) DedupInPlace

func (util *TableUtil) DedupInPlace(src string)

DedupInPlace executes a query that dedups a table. TODO interpret and return status.

func (*TableUtil) DestinationQuery

func (util *TableUtil) DestinationQuery(query string, dest *bigquery.Table) *bigquery.Query

DestinationQuery constructs a query with common Config settings for writing results to a table. Generally, may need to change WriteDisposition.

func (*TableUtil) GetInfoMatching

func (util *TableUtil) GetInfoMatching(dataset, filter string) []TableInfo

GetInfoMatching finDataset all tables matching table filter. and collects the basic stats about each of them. Returns slice ordered by decreasing age.

func (*TableUtil) GetTableStats

func (util *TableUtil) GetTableStats(table string) bigquery.TableMetadata

GetTableStats fetches the Metadata for a table. TODO(gfr) Is this worth having, or is it non-idiomatic?

func (*TableUtil) QueryAndParse

func (util *TableUtil) QueryAndParse(q string, model interface{}) (interface{}, error)

QueryAndParse executes a query that should return a single row, with column labels matching the qfields tags in the provided model struct.

func (*TableUtil) ResultQuery

func (util *TableUtil) ResultQuery(query string, dryRun bool) *bigquery.Query

ResultQuery constructs a query with common Config settings for writing results to a table. Generally, may need to change WriteDisposition.