bq

Documentation

Overview

Package bq provides a wrapper for Google's BigQuery library as well as general setup of the BigQuery client and streaming inserts usign bqstreamer.

Index

• func Close()
• func EncodeLegacy(v interface{}, omitEmpty bool) (map[string]bigquery.JsonValue, error)
• func EnsureTable(table *bigquery.Table, schema bigquery.Schema, ...)
• func IsSameSchema(s1, s2 bigquery.Schema) bool
• func SetupStreamingInserts()
• type BigQueryWrapper
  • func NewBigQueryWrapper(client *bigquery.Client, projectId, datasetId string) *BigQueryWrapper
  • func Setup() *BigQueryWrapper
  • func (w *BigQueryWrapper) AddRow(tableId string, row interface{}) error
  • func (w *BigQueryWrapper) Dataset() *bigquery.Dataset
  • func (w *BigQueryWrapper) Table(tableId string) *bigquery.Table
  • func (w *BigQueryWrapper) TableId(tableId string) string
  • func (w *BigQueryWrapper) UseTablePrefix(useIt bool)

Functions

func Close

func Close()

Close any open workers.

func EncodeLegacy

func EncodeLegacy(v interface{}, omitEmpty bool) (map[string]bigquery.JsonValue, error)

Encode takes a struct and returns a BigQuery compatible encoded map for the legacy BigQuery library.

func EnsureTable

func EnsureTable(table *bigquery.Table, schema bigquery.Schema, extraMeta *bigquery.TableMetadata)

Ensure that the given table is up-to-date. - If the table does not exist, it will be created - If the table exists but the schema is outdated, it will be updated - Otherwise nothing happens The function panics if any of its API calls fail.

func IsSameSchema

func IsSameSchema(s1, s2 bigquery.Schema) bool

Determines whether two BigQuery schemas are the same. Currently does not support nested schemas and only checks the name and type of the fields.

func SetupStreamingInserts

func SetupStreamingInserts()

Sets up workers for handling streaming inserts using multiple concurrent go routines. Expects the same environment variables as Setup().

Types

type BigQueryWrapper

type BigQueryWrapper struct {
	Client      *bigquery.Client
	ProjectId   string
	DatasetId   string
	TablePrefix string
}

func NewBigQueryWrapper

func NewBigQueryWrapper(client *bigquery.Client, projectId, datasetId string) *BigQueryWrapper

Creates a new wrapper for a bigquery client

func Setup

func Setup() *BigQueryWrapper

Sets up the BigQuery client wrapper. Assumes that the following environment variables are set: BIGQUERY_PROJECT_ID: The ID of the bigquery project, from the Google Cloud console BIGQUERY_DATASET_ID: The ID of dataset to use. BIGQUERY_SERVICE_ACCOUNT: A filepath for a bigquery service account configuration OR a base64 encoded string with the service account credentials. The created wrapper is returned.

func (*BigQueryWrapper) AddRow

func (w *BigQueryWrapper) AddRow(tableId string, row interface{}) error

func (*BigQueryWrapper) Dataset

func (w *BigQueryWrapper) Dataset() *bigquery.Dataset

func (*BigQueryWrapper) Table

func (w *BigQueryWrapper) Table(tableId string) *bigquery.Table

func (*BigQueryWrapper) TableId

func (w *BigQueryWrapper) TableId(tableId string) string

func (*BigQueryWrapper) UseTablePrefix

func (w *BigQueryWrapper) UseTablePrefix(useIt bool)