Back to

package pipelinex

Latest Go to latest
Published: Jun 3, 2020 | License: Apache-2.0 | Module:


Package pipelinex contains utilities for manipulating Beam proto pipelines. The utilities generally uses shallow copies and do not mutate their inputs.


func Bounded

func Bounded(p *pipepb.Pipeline) bool

Bounded returns true iff all PCollections are bounded.

func ContainerImages

func ContainerImages(p *pipepb.Pipeline) []string

ContainerImages returns the set of container images used in the given pipeline.

func Normalize

func Normalize(p *pipepb.Pipeline) (*pipepb.Pipeline, error)

Normalize recomputes derivative information in the pipeline, such as roots and input/output for composite transforms. It also ensures that unique names are so and topologically sorts each subtransform list.

func ShallowCloneFunctionSpec

func ShallowCloneFunctionSpec(p *pb.FunctionSpec) *pb.FunctionSpec

ShallowCloneFunctionSpec makes a shallow copy of the given FunctionSpec.

func ShallowClonePTransform

func ShallowClonePTransform(t *pb.PTransform) *pb.PTransform

ShallowClonePTransform makes a shallow copy of the given PTransform.

func ShallowCloneParDoPayload

func ShallowCloneParDoPayload(p *pb.ParDoPayload) *pb.ParDoPayload

ShallowCloneParDoPayload makes a shallow copy of the given ParDoPayload.

func ShallowCloneSideInput

func ShallowCloneSideInput(p *pb.SideInput) *pb.SideInput

ShallowCloneSideInput makes a shallow copy of the given SideInput.

func TopologicalSort

func TopologicalSort(xforms map[string]*pipepb.PTransform, ids []string) []string

TopologicalSort returns a topologically sorted list of the given ids, generally from the same scope/composite. Assumes acyclic graph.

func TrimCoders

func TrimCoders(coders map[string]*pipepb.Coder, ids ...string) map[string]*pipepb.Coder

TrimCoders returns the transitive closure of the given coders ids.

func Update

func Update(p *pipepb.Pipeline, values *pipepb.Components) (*pipepb.Pipeline, error)

Update merges a pipeline with the given components, which may add, replace or delete its values. It returns the merged pipeline. The input is not modified.

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
f or F : Jump to identifier