sql

Documentation

Overview

Package sql contains SQL transform APIs, allowing SQL queries to be used in Beam Go pipelines.

NOTE: This is an experimental feature. It currently only works when an expansion service/handler is registered for SQL transform. The APIs are subject to change without backward compatibility guarantees.

Index

• func Transform(s beam.Scope, query string, opts ...Option) beam.PCollection
• type Option
  • func Dialect(dialect string) Option
  • func ExpansionAddr(addr string) Option
  • func Input(name string, in beam.PCollection) Option
  • func OutputType(t reflect.Type, components ...typex.FullType) Option

Functions

func Transform

func Transform(s beam.Scope, query string, opts ...Option) beam.PCollection

Transform creates a SQL-based transform over zero or more PCollections and/or named data sources.

PCollection inputs can be added using the sql.Input option. SQL queries can also refer to external tables that can be resolved by the expansion service.

The output PCollection type must be specified by the sql.OutputType option.

Example:

in := beam.Create(s, 1, 2, 3)
out := sql.Transform(s, "SELECT COUNT(*) FROM t",
    sql.Input("t", in),
    sql.OutputType(reflect.TypeOf(int64(0))))
// `out` is a PCollection<int64> with a single element 3.

If an expansion service address is not provided as an option, one will be automatically started for the transform.

Types

type Option

type Option func(sqlx.Options)

Option is the base type of all the SQL transform options.

func Dialect

func Dialect(dialect string) Option

Dialect specifies the SQL dialect, e.g. use 'zetasql' for ZetaSQL.

func ExpansionAddr

func ExpansionAddr(addr string) Option

ExpansionAddr is the URL of the expansion service to use.

func Input

func Input(name string, in beam.PCollection) Option

Input adds a named PCollection input to the transform.

func OutputType

func OutputType(t reflect.Type, components ...typex.FullType) Option

OutputType specifies the output PCollection type of the transform. It must match the SQL output schema.

There is currently no default output type, so users must set this option. In the future, Row, once implemented, may become the default output type.