snippets

package
v2.55.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 2, 2024 License: Apache-2.0, BSD-3-Clause, MIT Imports: 23 Imported by: 0

Documentation

Overview

Package snippets contains code used in the Beam Programming Guide as examples for the Apache Beam Go SDK. These snippets are compiled and their tests run to ensure correctness. However, due to their piecemeal pedagogical use, they may not be the best example of production code.

The Beam Programming Guide can be found at https://beam.apache.org/documentation/programming-guide/.

Package snippets contains code used in the Beam Programming Guide as examples for the Apache Beam Go SDK. These snippets are compiled and their tests run to ensure correctness. However, due to their piecemeal pedagogical use, they may not be the best example of production code.

The Beam Programming Guide can be found at https://beam.apache.org/documentation/programming-guide/.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddBufferDoFn added in v2.50.0 

func AddBufferDoFn[V any](s beam.Scope, in beam.PCollection) beam.PCollection

func AddDynamicTimerTagsDoFn added in v2.50.0 

func AddDynamicTimerTagsDoFn[V hasAction](s beam.Scope, in beam.PCollection) beam.PCollection

func AddEventTimeDoFn added in v2.50.0 

func AddEventTimeDoFn(s beam.Scope, in beam.PCollection) beam.PCollection

func AddJoinDoFn added in v2.50.0 

func AddJoinDoFn(s beam.Scope, in beam.PCollection) beam.PCollection

func AddProcessingTimeDoFn added in v2.50.0 

func AddProcessingTimeDoFn(s beam.Scope, in beam.PCollection) beam.PCollection

func AddTimedOutputBatching added in v2.50.0 

func AddTimedOutputBatching[V any](s beam.Scope, in beam.PCollection) beam.PCollection

func AddTimerGarbageCollection added in v2.50.0 

func AddTimerGarbageCollection[V any](s beam.Scope, in beam.PCollection) beam.PCollection

func AddTimestampDoFn added in v2.34.0 

func AddTimestampDoFn(element LogEntry, emit func(beam.EventTime, LogEntry))

AddTimestampDoFn extracts an event time from a LogEntry.

func ComplexTriggers added in v2.34.0 

func ComplexTriggers(s beam.Scope, pcollection beam.PCollection)

func CountWords 

func CountWords(s beam.Scope, lines beam.PCollection) beam.PCollection

[START countwords_composite] CountWords is a function that builds a composite PTransform to count the number of times each word appears.

func Create 

func Create()

Create demonstrates using beam.CreateList.

func CreateAndSplit 

func CreateAndSplit(s beam.Scope, input []stringPair) beam.PCollection

CreateAndSplit is a helper function that creates

func LogicalTypeExample added in v2.34.0 

func LogicalTypeExample()

func PipelineConstruction 

func PipelineConstruction()

PipelineConstruction contains snippets for the initial sections of the Beam Programming Guide, from initializing to submitting a pipeline.

func PipelineOptions 

func PipelineOptions()

PipelineOptions shows basic pipeline options using flags.

func PipelineOptionsCustom 

func PipelineOptionsCustom()

PipelineOptionsCustom shows slightly less basic pipeline options using flags.

func TriggerAfterEndOfWindow added in v2.34.0 

func TriggerAfterEndOfWindow(s beam.Scope, pCollection beam.PCollection)

func TriggerAlways added in v2.34.0 

func TriggerAlways(s beam.Scope, pCollection beam.PCollection)

Types

type ComputeWordLengthFn 

type ComputeWordLengthFn struct{}

ComputeWordLengthFn is the DoFn to perform on each element in the input PCollection.

func (*ComputeWordLengthFn) ProcessElement 

func (fn *ComputeWordLengthFn) ProcessElement(word string, emit func(int))

ProcessElement is the method to execute for each element.

type CustomWatermarkEstimator added in v2.39.0 

type CustomWatermarkEstimator struct {
	// contains filtered or unexported fields
}

CustomWatermarkEstimator is a custom watermark estimator. You may use any type here, including some of Beam's built in watermark estimator types, e.g. sdf.WallTimeWatermarkEstimator, sdf.TimestampObservingWatermarkEstimator, and sdf.ManualWatermarkEstimator

func (*CustomWatermarkEstimator) CurrentWatermark added in v2.39.0 

func (e *CustomWatermarkEstimator) CurrentWatermark() time.Time

CurrentWatermark returns the current watermark and is invoked on DoFn splits and self-checkpoints. Watermark estimators must implement CurrentWatermark() time.Time

func (*CustomWatermarkEstimator) ObserveTimestamp added in v2.39.0 

func (e *CustomWatermarkEstimator) ObserveTimestamp(ts time.Time)

ObserveTimestamp is called on the output timestamps of all emitted elements to update the watermark. It is optional

type Event added in v2.50.0 

type Event struct{}

type JoinedEvent added in v2.50.0 

type JoinedEvent struct {
	View, Click *Event
}

type LogEntry added in v2.34.0 

type LogEntry int

LogEntry is a dummy type for documentation purposes.

type MyCustomType added in v2.43.0 

type MyCustomType struct{}

func (MyCustomType) Bytes added in v2.43.0 

func (m MyCustomType) Bytes() []byte

func (MyCustomType) FromBytes added in v2.43.0 

func (m MyCustomType) FromBytes(_ []byte) MyCustomType

type MyMetricsDoFn added in v2.34.0 

type MyMetricsDoFn struct {
	// contains filtered or unexported fields
}

func (*MyMetricsDoFn) ProcessElement added in v2.34.0 

func (fn *MyMetricsDoFn) ProcessElement(ctx context.Context, v beam.V, emit func(beam.V))

func (*MyMetricsDoFn) Setup added in v2.34.0 

func (fn *MyMetricsDoFn) Setup()

type Purchase added in v2.34.0 

type Purchase struct {
	// ID of the user who made the purchase.
	UserID string `beam:"userId"`
	// Identifier of the item that was purchased.
	ItemID int64 `beam:"itemId"`
	// The shipping address, a nested type.
	ShippingAddress ShippingAddress `beam:"shippingAddress"`
	// The cost of the item in cents.
	Cost int64 `beam:"cost"`
	// The transactions that paid for this purchase.
	// A slice since the purchase might be spread out over multiple
	// credit cards.
	Transactions []Transaction `beam:"transactions"`
}

type Record added in v2.40.0 

type Record struct{}

type ShippingAddress added in v2.34.0 

type ShippingAddress struct {
	StreetAddress string  `beam:"streetAddress"`
	City          string  `beam:"city"`
	State         *string `beam:"state"`
	Country       string  `beam:"country"`
	PostCode      string  `beam:"postCode"`
}

type SomeService added in v2.40.0 

type SomeService struct {
	ThrottlingErr error
}

type Student 

type Student struct {
	Percentile int
}

type TimestampNanos added in v2.34.0 

type TimestampNanos time.Time

TimestampNanos is a logical type using time.Time, but encodes as a schema type.

func (TimestampNanos) Nanos added in v2.34.0 

func (tn TimestampNanos) Nanos() int32

func (TimestampNanos) Seconds added in v2.34.0 

func (tn TimestampNanos) Seconds() int64

type TimestampNanosProvider added in v2.34.0 

type TimestampNanosProvider struct{}

TimestampNanosProvider implements the beam.SchemaProvider interface.

func (*TimestampNanosProvider) BuildDecoder added in v2.34.0 

func (p *TimestampNanosProvider) BuildDecoder(rt reflect.Type) (func(io.Reader) (any, error), error)

BuildDecoder builds a Beam schema decoder for the TimestampNanos type.

func (*TimestampNanosProvider) BuildEncoder added in v2.34.0 

func (p *TimestampNanosProvider) BuildEncoder(rt reflect.Type) (func(any, io.Writer) error, error)

BuildEncoder builds a Beam schema encoder for the TimestampNanos type.

func (*TimestampNanosProvider) FromLogicalType added in v2.34.0 

func (p *TimestampNanosProvider) FromLogicalType(rt reflect.Type) (reflect.Type, error)

FromLogicalType converts checks if the given type is TimestampNanos, and if so returns the storage type.

type Transaction added in v2.34.0 

type Transaction struct {
	Bank           string  `beam:"bank"`
	PurchaseAmount float64 `beam:"purchaseAmount"`
}

type WatermarkState added in v2.39.0 

type WatermarkState struct {
	Watermark time.Time
}

WatermarkState is a custom type.`

It is optional to write your own state type when making a custom estimator.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.