bqutil

Documentation

Overview

Package bqutil provides utility functions to interact with BigQuery.

Index

• Constants
• func Client(ctx context.Context, gcpProject string) (*bigquery.Client, error)
• func DatasetForProject(luciProject string) (string, error)
• func FatalError(err error) bool
• func GenerateSchema(fdset *desc.FileDescriptorSet, message string) (schema bigquery.Schema, err error)
• func NewWriterClient(ctx context.Context, gcpProject string) (*managedwriter.Client, error)
• func ProjectForDataset(dataset string) (string, error)
• func WaitForJob(ctx context.Context, job *bigquery.Job) (*bigquery.JobStatus, error)
• type Inserter
  • func NewInserter(table *bigquery.Table, batchSize int) *Inserter
  • func (i *Inserter) Put(ctx context.Context, rows []*bq.Row) error
  • func (i *Inserter) PutWithRetries(ctx context.Context, rows []*bq.Row) error
• type Writer
  • func NewWriter(client *managedwriter.Client, tableName string, ...) *Writer
  • func (s *Writer) AppendRowsWithDefaultStream(ctx context.Context, rows []proto.Message) error
  • func (s *Writer) AppendRowsWithPendingStream(ctx context.Context, rows []proto.Message) error

Constants

const InternalDatasetID = "internal"

InternalDatasetID is the name of the BigQuery dataset which is intended for internal service use only.

Functions

func Client

func Client(ctx context.Context, gcpProject string) (*bigquery.Client, error)

Client returns a new BigQuery client for use with the given GCP project, that authenticates as LUCI Analysis itself. Only use this method if the specification of the BigQuery dataset to access is not under the control of the project (e.g. via configuration).

func DatasetForProject

func DatasetForProject(luciProject string) (string, error)

DatasetForProject returns the name of the BigQuery dataset that contains the given project's data, in the LUCI Analysis GCP project.

func FatalError

func FatalError(err error) bool

FatalError returns true if the error is a known fatal error.

func GenerateSchema

func GenerateSchema(fdset *desc.FileDescriptorSet, message string) (schema bigquery.Schema, err error)

GenerateSchema generates BigQuery schema for the given proto message using the given set of message definitions.

func NewWriterClient

func NewWriterClient(ctx context.Context, gcpProject string) (*managedwriter.Client, error)

NewWriterClient returns a new BigQuery managedwriter client for use with the given GCP project, that authenticates as LUCI Analysis itself.

func ProjectForDataset

func ProjectForDataset(dataset string) (string, error)

ProjectForDataset returns the name of the LUCI Project that corresponds to the given BigQuery dataset.

func WaitForJob

func WaitForJob(ctx context.Context, job *bigquery.Job) (*bigquery.JobStatus, error)

WaitForJob waits for a BigQuery job to finish. If after timeout and the job has not finished, it will attempt to cancel the job. The cancellation is based on best-effort, so if there is an error, we just log instead of throwing the error. This is to avoid jobs overrunning each other and triggering a death spiral of write contention / starving each other of resources. The actual timeout for bigquery job will be context timeout reduced by 5 seconds. It is for the cancelling job to execute. If the context does not have a deadline, the bigquery job will have no timeout.

Types

type Inserter

type Inserter struct {
	// contains filtered or unexported fields
}

Inserter provides methods to insert rows into a BigQuery table.

func NewInserter

func NewInserter(table *bigquery.Table, batchSize int) *Inserter

NewInserter initialises a new inserter.

func (*Inserter) Put

func (i *Inserter) Put(ctx context.Context, rows []*bq.Row) error

Put inserts the given rows into BigQuery.

func (*Inserter) PutWithRetries

func (i *Inserter) PutWithRetries(ctx context.Context, rows []*bq.Row) error

PutWithRetries puts rows into BigQuery. Retries on transient errors.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is used to export rows to BigQuery table.

func NewWriter

func NewWriter(
	client *managedwriter.Client,
	tableName string,
	tableSchemaDescriptor *descriptorpb.DescriptorProto,
) *Writer

NewWriter creates a writer for exporting rows to the provided BigQuery table via the provided managedWriter client.

func (*Writer) AppendRowsWithDefaultStream

func (s *Writer) AppendRowsWithDefaultStream(ctx context.Context, rows []proto.Message) error

AppendRowsWithDefaultStream write to the default stream. This does not provide exactly-once semantics (it provides at leas once). The at least once semantic is similar to the legacy streaming API.

func (*Writer) AppendRowsWithPendingStream

func (s *Writer) AppendRowsWithPendingStream(ctx context.Context, rows []proto.Message) error

AppendRowsWithPendingStream append rows to BigQuery table via the pending stream. This provides all-or-nothing semantics for insertion.