Documentation ¶
Overview ¶
Package bigquery provides a client for the BigQuery service.
Note: This package is a work-in-progress. Backwards-incompatible changes should be expected.
Index ¶
- Constants
- type Client
- func (c *Client) Copy(ctx context.Context, dst Destination, src Source, options ...Option) (*Job, error)
- func (c *Client) CreateTable(ctx context.Context, projectID, datasetID, tableID string, ...) (*Table, error)
- func (c *Client) Dataset(id string) *Dataset
- func (c *Client) JobFromID(ctx context.Context, id string) (*Job, error)
- func (c *Client) NewGCSReference(uri ...string) *GCSReference
- func (c *Client) OpenTable(projectID, datasetID, tableID string) *Table
- func (c *Client) Read(ctx context.Context, src ReadSource, options ...ReadOption) (*Iterator, error)
- type Compression
- type CreateTableOption
- type DataFormat
- type Dataset
- type Destination
- type Encoding
- type Error
- type FieldSchema
- type FieldType
- type GCSReference
- type Iterator
- type Job
- type JobStatus
- type Option
- func AllowJaggedRows() Option
- func AllowQuotedNewlines() Option
- func CreateDisposition(disp TableCreateDisposition) Option
- func DestinationSchema(schema Schema) Option
- func DisableHeader() Option
- func DisableQueryCache() Option
- func IgnoreUnknownValues() Option
- func JobID(ID string) Option
- func JobPriority(priority string) Option
- func MaxBadRecords(n int64) Option
- func WriteDisposition(disp TableWriteDisposition) Option
- type Query
- type ReadOption
- type ReadSource
- type Schema
- type Source
- type State
- type Table
- type TableCreateDisposition
- type TableMetadata
- type TableMetadataPatch
- type TableType
- type TableWriteDisposition
- type Tables
- type Value
- type ValueList
- type ValueLoader
Constants ¶
const ( BatchPriority = "BATCH" InteractivePriority = "INTERACTIVE" )
const Scope = "https://www.googleapis.com/auth/bigquery"
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client may be used to perform BigQuery operations.
func NewClient ¶
NewClient constructs a new Client which can perform BigQuery operations. Operations performed via the client are billed to the specified GCP project. The supplied http.Client is used for making requests to the BigQuery server and must be capable of authenticating requests with Scope.
func (*Client) Copy ¶
func (c *Client) Copy(ctx context.Context, dst Destination, src Source, options ...Option) (*Job, error)
Copy starts a BigQuery operation to copy data from a Source to a Destination.
func (*Client) CreateTable ¶
func (c *Client) CreateTable(ctx context.Context, projectID, datasetID, tableID string, options ...CreateTableOption) (*Table, error)
CreateTable creates a table in the BigQuery service and returns a handle to it.
func (*Client) JobFromID ¶
JobFromID creates a Job which refers to an existing BigQuery job. The job need not have been created by this package. For example, the job may have been created in the BigQuery console.
func (*Client) NewGCSReference ¶
func (c *Client) NewGCSReference(uri ...string) *GCSReference
NewGCSReference constructs a reference to one or more Google Cloud Storage objects, which together constitute a data source or destination. In the simple case, a single URI in the form gs://bucket/object may refer to a single GCS object. Data may also be split into mutiple files, if multiple URIs or URIs containing wildcards are provided. Each URI may contain one '*' wildcard character, which (if present) must come after the bucket name. For more information about the treatment of wildcards and multiple URIs, see https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple
func (*Client) OpenTable ¶
OpenTable creates a handle to an existing BigQuery table. If the table does not already exist, subsequent uses of the *Table will fail.
func (*Client) Read ¶
func (c *Client) Read(ctx context.Context, src ReadSource, options ...ReadOption) (*Iterator, error)
Read fetches data from a ReadSource and returns the data via an Iterator.
type Compression ¶
type Compression string
Compression is the type of compression to apply when writing data to Google Cloud Storage.
const ( None Compression = "NONE" Gzip Compression = "GZIP" )
type CreateTableOption ¶
type CreateTableOption interface {
// contains filtered or unexported methods
}
A CreateTableOption is an optional argument to CreateTable.
func TableExpiration ¶
func TableExpiration(exp time.Time) CreateTableOption
TableExpiration returns a CreateTableOption which will cause the created table to be deleted after the expiration time.
func ViewQuery ¶
func ViewQuery(query string) CreateTableOption
ViewQuery returns a CreateTableOption that causes the created table to be a virtual table defined by the supplied query. For more information see: https://cloud.google.com/bigquery/querying-data#views
type DataFormat ¶
type DataFormat string
const ( CSV DataFormat = "CSV" Avro DataFormat = "AVRO" JSON DataFormat = "NEWLINE_DELIMITED_JSON" DatastoreBackup DataFormat = "DATASTORE_BACKUP" )
type Dataset ¶
type Dataset struct {
// contains filtered or unexported fields
}
Dataset is a reference to a BigQuery dataset.
type Destination ¶
type Destination interface {
// contains filtered or unexported methods
}
A Destination is a destination of data for the Copy function.
type Encoding ¶
type Encoding string
Encoding specifies the character encoding of data to be loaded into BigQuery. See https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.encoding for more details about how this is used.
type Error ¶
type Error struct {
// Mirrors bq.ErrorProto, but drops DebugInfo
Location, Message, Reason string
}
An Error contains detailed information about an error encountered while processing a job.
type FieldSchema ¶
type FieldSchema struct { // The field name. // Must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), // and must start with a letter or underscore. // The maximum length is 128 characters. Name string // A description of the field. The maximum length is 16,384 characters. Description string // Whether the field may contain multiple values. Repeated bool // Whether the field is required. Ignored if Repeated is true. Required bool // The field data type. If Type is Record, then this field contains a nested schema, // which is described by Schema. Type FieldType // Describes the nested schema if Type is set to Record. Schema Schema }
type GCSReference ¶
type GCSReference struct { // FieldDelimiter is the separator for fields in a CSV file, used when loading or exporting data. // The default is ",". FieldDelimiter string // The number of rows at the top of a CSV file that BigQuery will skip when loading the data. SkipLeadingRows int64 // SourceFormat is the format of the GCS data to be loaded into BigQuery. // Allowed values are: CSV, JSON, DatastoreBackup. The default is CSV. SourceFormat DataFormat // Only used when loading data. Encoding Encoding // Quote is the value used to quote data sections in a CSV file. // The default quotation character is the double quote ("), which is used if both Quote and ForceZeroQuote are unset. // To specify that no character should be interpreted as a quotation character, set ForceZeroQuote to true. // Only used when loading data. Quote string ForceZeroQuote bool // DestinationFormat is the format to use when writing exported files. // Allowed values are: CSV, Avro, JSON. The default is CSV. // CSV is not supported for tables with nested or repeated fields. DestinationFormat DataFormat // Only used when writing data. Default is None. Compression Compression // contains filtered or unexported fields }
GCSReference is a reference to one or more Google Cloud Storage objects, which together constitute an input or output to a BigQuery operation.
type Iterator ¶
type Iterator struct {
// contains filtered or unexported fields
}
Iterator provides access to the result of a BigQuery lookup. Next must be called before the first call to Get.
func (*Iterator) Next ¶
Next advances the Iterator to the next row, making that row available via the Get method. Next must be called before the first call to Get, and blocks until data is available. Next returns false when there are no more rows available, either because the end of the output was reached, or because there was an error (consult the Err method to determine which).
type Job ¶
type Job struct {
// contains filtered or unexported fields
}
A Job represents an operation which has been submitted to BigQuery for processing.
type JobStatus ¶
type JobStatus struct { State State // All errors encountered during the running of the job. // Not all Errors are fatal, so errors here do not necessarily mean that the job has completed or was unsuccessful. Errors []*Error // contains filtered or unexported fields }
JobStatus contains the current State of a job, and errors encountered while processing that job.
type Option ¶
type Option interface {
// contains filtered or unexported methods
}
An Option is an optional argument to Copy.
func AllowJaggedRows ¶
func AllowJaggedRows() Option
AllowJaggedRows returns an Option that causes missing trailing optional columns to be tolerated in CSV data. Missing values are treated as nulls.
func AllowQuotedNewlines ¶
func AllowQuotedNewlines() Option
AllowQuotedNewlines returns an Option that allows quoted data sections containing newlines in CSV data.
func CreateDisposition ¶
func CreateDisposition(disp TableCreateDisposition) Option
func DestinationSchema ¶
DestinationSchema returns an Option that specifies the schema to use when loading data into a new table. A DestinationSchema Option must be supplied when loading data from Google Cloud Storage into a non-existent table. Caveat: DestinationSchema is not required if the data being loaded is a datastore backup. schema must not be nil.
func DisableHeader ¶
func DisableHeader() Option
DisableHeader returns an Option that disables the printing of a header row in exported data.
func DisableQueryCache ¶
func DisableQueryCache() Option
DisableQueryCache returns an Option that prevents results being fetched from the query cache. If this Option is not used, results are fetched from the cache if they are available. The query cache is a best-effort cache that is flushed whenever tables in the query are modified. Cached results are only available when TableID is unspecified in the query's destination Table. For more information, see https://cloud.google.com/bigquery/querying-data#querycaching
func IgnoreUnknownValues ¶
func IgnoreUnknownValues() Option
IgnoreUnknownValues returns an Option that causes values not matching the schema to be tolerated. Unknown values are ignored. For CSV this ignores extra values at the end of a line. For JSON this ignores named values that do not match any column name. If this Option is not used, records containing unknown values are treated as bad records. The MaxBadRecords Option can be used to customize how bad records are handled.
func JobID ¶
JobID returns an Option that sets the job ID of a BigQuery job. If this Option is not used, a job ID is generated automatically.
func JobPriority ¶
JobPriority returns an Option that causes a query to be scheduled with the specified priority. The default priority is InteractivePriority. For more information, see https://cloud.google.com/bigquery/querying-data#batchqueries
func MaxBadRecords ¶
MaxBadRecords returns an Option that sets the maximum number of bad records that will be ignored. If this maximum is exceeded, the operation will be unsuccessful.
func WriteDisposition ¶
func WriteDisposition(disp TableWriteDisposition) Option
type Query ¶
type Query struct { // The query to execute. See https://cloud.google.com/bigquery/query-reference for details. Q string // DefaultProjectID and DefaultDatasetID specify the dataset to use for unqualified table names in the query. // If DefaultProjectID is set, DefaultDatasetID must also be set. DefaultProjectID string DefaultDatasetID string }
Query represents a query to be executed.
type ReadOption ¶
type ReadOption interface {
// contains filtered or unexported methods
}
A ReadOption is an optional argument to Read.
func RecordsPerRequest ¶
func RecordsPerRequest(n int64) ReadOption
RecordsPerRequest returns a ReadOption that sets the number of records to fetch per request when streaming data from BigQuery.
func StartIndex ¶
func StartIndex(i uint64) ReadOption
StartIndex returns a ReadOption that sets the zero-based index of the row to start reading from.
type ReadSource ¶
type ReadSource interface {
// contains filtered or unexported methods
}
A ReadSource is a source of data for the Read function.
type Source ¶
type Source interface {
// contains filtered or unexported methods
}
A Source is a source of data for the Copy function.
type State ¶
type State int
State is one of a sequence of states that a Job progresses through as it is processed.
type Table ¶
type Table struct { // ProjectID, DatasetID and TableID may be omitted if the Table is the destination for a query. // In this case the result will be stored in an ephemeral table. ProjectID string DatasetID string // TableID must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_). // The maximum length is 1,024 characters. TableID string // contains filtered or unexported fields }
A Table is a reference to a BigQuery table.
func (*Table) FullyQualifiedName ¶
FullyQualifiedName returns the ID of the table in projectID:datasetID.tableID format.
func (*Table) Metadata ¶
func (t *Table) Metadata(ctx context.Context) (*TableMetadata, error)
Metadata fetches the metadata for the table.
func (*Table) Patch ¶
func (t *Table) Patch() *TableMetadataPatch
Patch returns a *TableMetadataPatch, which can be used to modify specific Table metadata fields. In order to apply the changes, the TableMetadataPatch's Apply method must be called.
type TableCreateDisposition ¶
type TableCreateDisposition string
CreateDisposition specifies the circumstances under which destination table will be created. Default is CreateIfNeeded.
const ( // The table will be created if it does not already exist. Tables are created atomically on successful completion of a job. CreateIfNeeded TableCreateDisposition = "CREATE_IF_NEEDED" // The table must already exist and will not be automatically created. CreateNever TableCreateDisposition = "CREATE_NEVER" )
type TableMetadata ¶
type TableMetadata struct { Description string // The user-friendly description of this table. Name string // The user-friendly name for this table. Schema Schema View string ID string // An opaque ID uniquely identifying the table. Type TableType // The time when this table expires. If not set, the table will persist // indefinitely. Expired tables will be deleted and their storage reclaimed. ExpirationTime time.Time CreationTime time.Time LastModifiedTime time.Time // The size of the table in bytes. // This does not include data that is being buffered during a streaming insert. NumBytes int64 // The number of rows of data in this table. // This does not include data that is being buffered during a streaming insert. NumRows uint64 }
TableMetadata contains information about a BigQuery table.
type TableMetadataPatch ¶
type TableMetadataPatch struct {
// contains filtered or unexported fields
}
TableMetadataPatch represents a set of changes to a table's metadata.
func (*TableMetadataPatch) Apply ¶
func (p *TableMetadataPatch) Apply(ctx context.Context) (*TableMetadata, error)
Apply applies the patch operation.
func (*TableMetadataPatch) Description ¶
func (p *TableMetadataPatch) Description(desc string)
Description sets the table description.
func (*TableMetadataPatch) Name ¶
func (p *TableMetadataPatch) Name(name string)
Name sets the table name.
type TableWriteDisposition ¶
type TableWriteDisposition string
TableWriteDisposition specifies how existing data in a destination table is treated. Default is WriteAppend.
const ( // Data will be appended to any existing data in the destination table. // Data is appended atomically on successful completion of a job. WriteAppend TableWriteDisposition = "WRITE_APPEND" // Existing data in the destination table will be overwritten. // Data is overwritten atomically on successful completion of a job. WriteTruncate TableWriteDisposition = "WRITE_TRUNCATE" // Writes will fail if the destination table already contains data. WriteEmpty TableWriteDisposition = "WRITE_EMPTY" )
type Tables ¶
type Tables []*Table
Tables is a group of tables. The tables may belong to differing projects or datasets.
type Value ¶
type Value interface{}
Value stores the contents of a single cell from a BigQuery result.
type ValueLoader ¶
ValueLoader stores a slice of Values representing a result row from a Read operation. See Iterator.Get for more information.