bigtableio

Documentation

Overview

Package bigtableio contains cross-language functionality for using Google Cloud BigQuery (https://cloud.google.com/bigquery). These transforms only work on runners that support cross-language transforms.

Setup

Transforms specified here are cross-language transforms implemented in a different SDK (listed below). During pipeline construction, the Go SDK will need to connect to an expansion service containing information on these transforms in their native SDK. If an expansion service address is not provided, an appropriate expansion service will be automatically started; however this is slower than having a persistent expansion service running.

To use a persistent expansion service, it must be run as a separate process accessible during pipeline construction. The address of that process must be passed to the transforms in this package.

The version of the expansion service should match the version of the Beam SDK being used. For numbered releases of Beam, these expansions services are released to the Maven repository as modules. For development versions of Beam, it is recommended to build and run it from source using Gradle.

Current supported SDKs, including expansion service modules and reference documentation:

Java:

• Vendored Module: beam-sdks-java-io-google-cloud-platform-expansion-service
• Run via Gradle: ./gradlew :sdks:java:io:google-cloud-platform:expansion-service:runExpansionService
• Reference Class: org.apache.beam.sdk.io.gcp.bigtable.BigtableReadSchemaTransformProvider and org.apache.beam.sdk.io.gcp.bigtable.BigtableIO

Note On Documentation

This cross-language implementation relies on the behavior of external SDKs. In order to keep documentation up-to-date and consistent, Bigtable functionality will not be described in detail in this package. Instead, references to relevant documentation in other SDKs is included where relevant.

Index

• func Read(s beam.Scope, projectId string, instanceId string, table string, ...) beam.PCollection
• type Cell
• type ReadOption
  • func ReadExpansionAddr(addr string) ReadOption
• type Row
  • func (row *Row) AddCell(family string, qualifier string, value []byte, timestamp int64)

Functions

func Read

func Read(s beam.Scope, projectId string, instanceId string, table string, opts ...ReadOption) beam.PCollection

Read reads rows from a Bigtable table.

Types

type Cell

type Cell = struct {
	Value            []uint8 `beam:"value"`
	Timestamp_micros int64   `beam:"timestamp_micros"`
}

Cell represents a single cell in a Bigtable row. TODO(https://github.com/apache/beam/issues/21784): Change back to a named struct once resolved.

type ReadOption

type ReadOption func(*readConfig)

func ReadExpansionAddr

func ReadExpansionAddr(addr string) ReadOption

ReadExpansionAddr specifies the address of a persistent expansion service to use for a Read transform. If this is not provided, or if an empty string is provided, the transform will automatically start an appropriate expansion service instead.

type Row

type Row struct {
	Key             []uint8                      `beam:"key"`
	Column_families map[string]map[string][]Cell `beam:"column_families"`
}

Row represents a row in Bigtable.

func (*Row) AddCell

func (row *Row) AddCell(family string, qualifier string, value []byte, timestamp int64)

AddCell adds cell to a Row. Note, this method does not deduplicate cells with the same (family, qualifier, timestamp) nor does it maintain cells sorted in timestamp-descending order.