v2.32.0+incompatible Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Aug 10, 2021 License: Apache-2.0, BSD-3-Clause, MIT Imports: 18 Imported by: 0



Package xlangx contains various low-level utilities needed for adding cross-language transforms to the pipeline.



This section is empty.


This section is empty.


func EncodeStructPayload

func EncodeStructPayload(pl interface{}) ([]byte, error)

EncodeStructPayload takes a native Go struct and returns a marshaled ExternalConfigurationPayload proto, containing a Schema representation of the original type and the original value encoded as a Row. This is intended to be used as the expansion payload for an External transform.

func Expand

func Expand(edge *graph.MultiEdge, ext *graph.ExternalTransform) error

Expand expands an unexpanded graph.ExternalTransform as a graph.ExpandedTransform and assigns it to the ExternalTransform's Expanded field. This requires querying an expansion service based on the configuration details within the ExternalTransform.

func ResolveArtifacts

func ResolveArtifacts(ctx context.Context, edges []*graph.MultiEdge, p *pipepb.Pipeline)

ResolveArtifacts acquires all dependencies for a cross-language transform

func ResolveArtifactsWithConfig

func ResolveArtifactsWithConfig(ctx context.Context, edges []*graph.MultiEdge, cfg ResolveConfig) (paths map[string]string, err error)

ResolveArtifactsWithConfig acquires all dependencies for cross-language transforms, but with some additional configuration to behavior. By default, this function performs the following steps for each cross-language transform in the list of edges:

  1. Retrieves a list of dependencies needed from the expansion service.
  2. Retrieves each dependency as an artifact and stages it to a default local filepath.
  3. Adds the dependencies to the transform's stored environment proto.

The changes that can be configured are documented in ResolveConfig.

This returns a map of "local path" to "sdk path". By default these are identical, unless ResolveConfig.SdkPath has been set.


type ResolveConfig

type ResolveConfig struct {
	// SdkPath replaces the default filepath for dependencies, but only in the
	// external environment proto to be used by the SDK Harness during pipeline
	// execution. This is used to specify alternate staging directories, such
	// as for staging artifacts remotely.
	// Setting an SdkPath does not change staging behavior otherwise. All
	// artifacts still get staged to the default local filepath, and it is the
	// user's responsibility to stage those local artifacts to the SdkPath.
	SdkPath string

	// JoinFn is a function for combining SdkPath and individual artifact names.
	// If not specified, it defaults to using filepath.Join.
	JoinFn func(path, name string) string

ResolveConfig contains fields for configuring the behavior for resolving artifacts.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL