Back to godoc.org

Package beam

v2.25.0+incompatible
Latest Go to latest

The latest major version is .

Published: 3 days ago | License: Apache-2.0 | Module: github.com/apache/beam
Path Synopsis
artifact Package artifact contains utilities for staging and retrieving artifacts.
artifact/gcsproxy Package gcsproxy contains artifact staging and retrieval servers backed by GCS.
core/funcx Package funcx contains functions and types used to perform type analysis of Beam functions.
core/graph Package graph is the internal representation of the Beam execution plan.
core/graph/coder Package coder contains coder representation and utilities.
core/graph/mtime Package mtime contains a millisecond representation of time.
core/graph/window Package window contains window representation, windowing strategies and utilities.
core/metrics Package metrics implements the Beam metrics API, described at http://s.apache.org/beam-metrics-api Metrics in the Beam model are uniquely identified by a namespace, a name, and the PTransform context in which they are used.
core/runtime Package runtime contains runtime hooks and utilities for pipeline options and type registration.
core/runtime/coderx Package coderx contains coders for primitive types that aren't included in the beam model.
core/runtime/exec Package exec contains runtime plan representation and execution.
core/runtime/exec/optimized Package optimized contains type-specialized shims for faster execution.
core/runtime/graphx Package graphx provides facilities to help with the serialization of pipelines into a serializable graph structure suitable for the worker.
core/runtime/graphx/schema Package schema contains utility functions for relating Go types and Beam Schemas.
core/runtime/graphx/v1 Package v1 is a generated protocol buffer package.
core/runtime/harness Package harness implements the SDK side of the Beam FnAPI.
core/runtime/harness/init Package init contains the harness initialization code defined by the FnAPI.
core/runtime/harness/session Package session is a generated protocol buffer package.
core/runtime/pipelinex Package pipelinex contains utilities for manipulating Beam proto pipelines.
core/runtime/xlangx
core/sdf Package contains interfaces used specifically for splittable DoFns.
core/typex Package typex contains full type representation for PCollections and DoFns, and utilities for type checking.
core/util/dot Package dot produces DOT graphs from Beam graph representations.
core/util/hooks Package hooks allows runners to tailor execution of the worker harness.
core/util/ioutilx Package ioutilx contains additional io utilities.
core/util/protox Package protox contains utilities for working with protobufs.
core/util/reflectx Package reflectx contains a set of reflection utilities and well-known types.
core/util/stringx Package stringx contains utilities for working with strings.
core/util/symtab Package symtab allows reading low-level symbol information from the symbol table.
internal/errors Package errors contains functionality for creating and wrapping errors with improved formatting compared to the standard Go error functionality.
io/avroio Package avroio contains transforms for reading and writing avro files.
io/bigqueryio Package bigqueryio provides transformations and utilities to interact with Google BigQuery.
io/databaseio Package databaseio provides transformations and utilities to interact with generic database database/sql API.
io/datastoreio Package datastoreio provides transformations and utilities to interact with Google Datastore.
io/filesystem Package filesystem contains an extensible file system abstraction.
io/filesystem/gcs Package gcs contains a Google Cloud Storage (GCS) implementation of the Beam file system.
io/filesystem/local Package local contains a local file implementation of the Beam file system.
io/filesystem/memfs Package memfs contains a in-memory Beam filesystem.
io/pubsubio Package pubsubio provides access to PubSub on Dataflow streaming.
io/pubsubio/v1 Package v1 is a generated protocol buffer package.
io/rtrackers/offsetrange Package offsetrange defines a restriction and restriction tracker for offset ranges.
io/synthetic Package synthetic contains transforms for creating synthetic pipelines.
io/textio Package textio contains transforms for reading and writing text files.
log Package log contains a re-targetable context-aware logging system.
model Package model contains the portable Beam model contracts.
model/fnexecution_v1
model/jobmanagement_v1
model/pipeline_v1
options/gcpopts Package gcpopts contains shared options for Google Cloud Platform.
options/jobopts Package jobopts contains shared options for job submission.
provision Package provision contains utilities for obtaining runtime provision, information -- such as pipeline options.
runners/dataflow Package dataflow contains the Dataflow runner for submitting pipelines to Google Cloud Dataflow.
runners/dataflow/dataflowlib Package dataflowlib translates a Beam pipeline model to the Dataflow API job model, for submission to Google Cloud Dataflow.
runners/direct Package direct contains the direct runner for running single-bundle pipelines in the current process.
runners/dot Package dot is a Beam runner that "runs" a pipeline by producing a DOT graph of the execution plan.
runners/flink Package flink contains the Flink runner.
runners/session
runners/spark Package spark contains the Spark runner.
runners/universal Package universal contains a general-purpose runner that can submit jobs to any portable Beam runner.
runners/universal/extworker Package extworker provides an external worker service and related utilities.
runners/universal/runnerlib Package runnerlib contains utilities for submitting Go pipelines to a Beam model runner.
runners/vet Package vet is a Beam runner that "runs" a pipeline by producing generated code to avoid symbol table lookups and reflection in pipeline execution.
runners/vet/testpipeline Package testpipeline exports small test pipelines for testing the vet runner.
testing/passert Package passert contains verification transformations for testing pipelines.
testing/ptest Package ptest contains utilities for pipeline unit testing.
transforms/filter Package filter contains transformations for removing pipeline elements based on various conditions.
transforms/stats Package stats contains transforms for statistical processing.
transforms/top Package top contains transformations for finding the smallest (or largest) N elements based on arbitrary orderings.
util/errorx Package errorx contains utilities for handling errors.
util/execx Package execx contains wrappers and utilities for the exec package.
util/gcsx Package gcsx contains utilities for working with Google Cloud Storage (GCS).
util/grpcx Package grpcx contains utilities for working with gRPC.
util/pubsubx Package pubsubx contains utilities for working with Google PubSub.
util/shimx Package shimx specifies the templates for generating type assertion shims for Apache Beam Go SDK pipelines.
util/starcgenx Package starcgenx is a Static Analysis Type Assertion shim and Registration Code Generator which provides an extractor to extract types from a package, in order to generate approprate shimsr a package so code can be generated for it.
util/syscallx Package syscallx provides system call utilities that attempt to hide platform differences.
x/beamx Package beamx is a convenience package for beam.
x/debug Package debug contains pipeline components that may help in debugging pipeline issues.
x/hooks/perf Package perf is to add performance measuring hooks to a runner, such as cpu, heap, or trace profiles.