nativepubsubio

Documentation

Overview

Package nativepubsubio contains a Golang implementation of streaming reads and writes to PubSub. This is not as fully featured as the cross-language pubsubio package present in the Beam Go repository and should not be used in place of it.

Index

• func Read(ctx context.Context, s beam.Scope, project, topic, subscription string) beam.PCollection
• func Write(ctx context.Context, s beam.Scope, col beam.PCollection, project, topic string)
• type SubscriptionRTracker
  • func NewSubscriptionRTracker(entry string) *SubscriptionRTracker
  • func (s *SubscriptionRTracker) GetError() error
  • func (s *SubscriptionRTracker) GetProgress() (done float64, remaining float64)
  • func (s *SubscriptionRTracker) GetRestriction() interface{}
  • func (s *SubscriptionRTracker) IsBounded() bool
  • func (s *SubscriptionRTracker) IsDone() bool
  • func (s *SubscriptionRTracker) TryClaim(pos interface{}) bool
  • func (s *SubscriptionRTracker) TrySplit(frac float64) (primary, residual interface{}, err error)

Functions

func Read

func Read(ctx context.Context, s beam.Scope, project, topic, subscription string) beam.PCollection

Read reads messages from a PubSub topic in a streaming context, outputting received messages as a PCollection of byte slices. If the provided subscription name exists for the given topic, the DoFn will read from that subscription; otherwise, a new subscription with the given subscription name will be created and read from.

This feature is experimental and subject to change, including its behavior and function signature. Please use the cross-language implementation Read() instead.

func Write

func Write(ctx context.Context, s beam.Scope, col beam.PCollection, project, topic string)

Write publishes elements from a PCollection of byte slices to a PubSub topic. If the topic does not exist at pipeline construction time, the function will panic.

This feature is experimental and subject to change, including its behavior and function signature. Please use the cross-language implementation Write() instead.

Types

type SubscriptionRTracker

type SubscriptionRTracker struct {
	Subscription string
	Done         bool
}

The SubscriptionRTracker maintains a single entry string to keep up with the PubSub subscription being used in the NativeRead SDF.

func NewSubscriptionRTracker

func NewSubscriptionRTracker(entry string) *SubscriptionRTracker

NewSubscriptionRTracker returns an RTracker wrapping the provided subscription and a "Done" boolean.

func (*SubscriptionRTracker) GetError

func (s *SubscriptionRTracker) GetError() error

GetError is a no-op.

func (*SubscriptionRTracker) GetProgress

func (s *SubscriptionRTracker) GetProgress() (done float64, remaining float64)

GetProgress returns complete just so the runner doesn't try to do much in the way of splitting.

func (*SubscriptionRTracker) GetRestriction

func (s *SubscriptionRTracker) GetRestriction() interface{}

GetRestriction returns the name of the subscription.

func (*SubscriptionRTracker) IsBounded

func (s *SubscriptionRTracker) IsBounded() bool

IsBounded always returns false, as the StaticRTracker represents an unbounded number of reads from PubSub.

func (*SubscriptionRTracker) IsDone

func (s *SubscriptionRTracker) IsDone() bool

IsDone returns whether or not the StaticRTracker is complete (e.g. has stopped processing.)

func (*SubscriptionRTracker) TryClaim

func (s *SubscriptionRTracker) TryClaim(pos interface{}) bool

TryClaim returns true iff the given position is a string and matches the underlying subscription ID.

func (*SubscriptionRTracker) TrySplit

func (s *SubscriptionRTracker) TrySplit(frac float64) (primary, residual interface{}, err error)

TrySplit is a no-op for the StaticRTracker in the normal case and moves the subscription to the residual in the checkpointing case, marking itself as done to keep the logical checks around SDF data loss happy.