Documentation

Overview

    Package runnerlib contains utilities for submitting Go pipelines to a Beam model runner.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    func BuildTempWorkerBinary

    func BuildTempWorkerBinary(ctx context.Context) (string, error)

      BuildTempWorkerBinary creates a local worker binary in the tmp directory for linux/amd64. Caller responsible for deleting the binary.

      func BuildWorkerBinary

      func BuildWorkerBinary(ctx context.Context, filename string) error

        BuildWorkerBinary creates a local worker binary for linux/amd64. It finds the filename by examining the call stack. We want the user entry (*), for example:

        /Users/herohde/go/src/github.com/apache/beam/sdks/go/pkg/beam/runners/beamexec/main.go (skip: 2)
        

        * /Users/herohde/go/src/github.com/apache/beam/sdks/go/examples/wordcount/wordcount.go (skip: 3)

        /usr/local/go/src/runtime/proc.go (skip: 4)      // not always present
        /usr/local/go/src/runtime/asm_amd64.s (skip: 4 or 5)
        

        func Execute

        func Execute(ctx context.Context, p *pipepb.Pipeline, endpoint string, opt *JobOptions, async bool) (string, error)

          Execute executes a pipeline on the universal runner serving the given endpoint. Convenience function.

          func IsWorkerCompatibleBinary

          func IsWorkerCompatibleBinary() (string, bool)

            IsWorkerCompatibleBinary returns the path to itself and true if running a linux-amd64 binary that can directly be used as a worker binary.

            func Prepare

            func Prepare(ctx context.Context, client jobpb.JobServiceClient, p *pipepb.Pipeline, opt *JobOptions) (id, endpoint, stagingToken string, err error)

              Prepare prepares a job to the given job service. It returns the preparation id artifact staging endpoint, and staging token if successful.

              func Stage

              func Stage(ctx context.Context, id, endpoint, binary, st string) (retrievalToken string, err error)

                Stage stages the worker binary and any additional files to the given artifact staging endpoint. It returns the retrieval token if successful.

                func StageFile

                func StageViaLegacyApi

                func StageViaLegacyApi(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retrievalToken string, err error)

                func StageViaPortableApi

                func StageViaPortableApi(ctx context.Context, cc *grpc.ClientConn, binary, st string) error

                func Submit

                func Submit(ctx context.Context, client jobpb.JobServiceClient, id, token string) (string, error)

                  Submit submits a job to the given job service. It returns a jobID, if successful.

                  func WaitForCompletion

                  func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error

                    WaitForCompletion monitors the given job until completion. It logs any messages and state changes received.

                    Types

                    type JobOptions

                    type JobOptions struct {
                    	// Name is the job name.
                    	Name string
                    	// Experiments are additional experiments.
                    	Experiments []string
                    
                    	// Worker is the worker binary override.
                    	Worker string
                    
                    	// RetainDocker is an option to pass to the runner.
                    	RetainDocker bool
                    }

                      JobOptions capture the various options for submitting jobs to universal runners.