pubsubio

Documentation

Overview

Package pubsubio provides access to Pub/Sub on Dataflow streaming.

This implementation only functions on the Dataflow runner.

See https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub for details on using Pub/Sub with Dataflow.

Index

• func Read(s beam.Scope, project string, opts ReadOptions) beam.PCollection
• func Write(s beam.Scope, project, topic string, col beam.PCollection)
• type ReadOptions

Functions

func Read

func Read(s beam.Scope, project string, opts ReadOptions) beam.PCollection

Read reads an unbounded number of PubSubMessages from the given pubsub topic or subscription. It produces an unbounded PCollecton<*PubSubMessage>, if WithAttributes is set, or an unbounded PCollection<[]byte>.

The topic or subscription is required and must be set with ReadOptions.

func Write

func Write(s beam.Scope, project, topic string, col beam.PCollection)

Write writes PubSubMessages or []bytes to the given pubsub topic. Panics if the input pcollection type is not one of those two types.

When given []bytes, they are first wrapped in PubSubMessages.

Note: Doesn't function in batch pipelines.

Types

type ReadOptions

type ReadOptions struct {
	Topic              string // Topic sets the topic to be read from. A new subscription will be generated for the job. Mutually exclusive with setting a Subscription.
	Subscription       string // Subscription sets the name of an existing subscription to read from. Mutually exclusive with setting a Topic.
	IDAttribute        string
	TimestampAttribute string
	WithAttributes     bool
}

ReadOptions represents options for reading from PubSub.