v0.5.28 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Mar 4, 2021 License: Apache-2.0 Imports: 14 Imported by: 0





View Source
const DefaultPhaseVersion = uint32(0)
View Source
const SystemErrorCode = "SystemError"


View Source
var PhaseInfoUndefined = PhaseInfo{/* contains filtered or unexported fields */}

PhaseInfoUndefined should be used when the Phase is unknown usually associated with an error

UnknownTransition is synonymous to UndefinedTransition. To be returned when an error is observed


This section is empty.


type AllocationStatus

type AllocationStatus int
const (
	// This is the enum returned when there's an error
	AllocationUndefined AllocationStatus = iota

	// Go for it

	// This means that no resources are available globally.  This is the only rejection message we use right now.

	// We're not currently using this - but this would indicate that things globally are okay, but that your
	// own namespace is too busy

func AllocationStatusString added in v0.5.23

func AllocationStatusString(s string) (AllocationStatus, error)

AllocationStatusString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func AllocationStatusValues added in v0.5.23

func AllocationStatusValues() []AllocationStatus

AllocationStatusValues returns all values of the enum

func (AllocationStatus) IsAAllocationStatus added in v0.5.23

func (i AllocationStatus) IsAAllocationStatus() bool

IsAAllocationStatus returns "true" if the value is listed in the enum definition. "false" otherwise

func (AllocationStatus) String added in v0.5.23

func (i AllocationStatus) String() string

type EnqueueOwner

type EnqueueOwner func(id types.NamespacedName) error

When a change is observed, the owning entity with id types.NamespacedName can be triggered for re-validation

type EventsRecorder

type EventsRecorder interface {
	RecordRaw(ctx context.Context, ev PhaseInfo) error

Task events recorder, which get stored in the Admin. If this is invoked multiple times, multiple events will be sent to Admin. It is not recommended that one uses this interface, a transition will trigger an auto event to admin

type KubeClient

type KubeClient interface {
	// GetClient returns a client configured with the Config
	GetClient() client.Client

	// GetCache returns a cache.Cache
	GetCache() cache.Cache

TODO we may not want to expose this? A friendly controller-runtime client that gets passed to executors

type Phase

type Phase int8
const (
	// Does not mean an error, but simply states that we dont know the state in this round, try again later. But can be used to signal a system error too
	PhaseUndefined Phase = iota
	// Indicates plugin is not ready to submit the request as it is waiting for resources
	// Indicates plugin has submitted the execution, but it has not started executing yet
	// The system has started the pre-execution process, like container download, cluster startup etc
	// Indicates that the task has started executing
	// Indicates that the task has completed successfully
	// Indicates that the Failure is recoverable, by re-executing the task if retries permit
	// Indicate that the failure is non recoverable even if retries exist

func PhaseString

func PhaseString(s string) (Phase, error)

PhaseString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func PhaseValues

func PhaseValues() []Phase

PhaseValues returns all values of the enum

func (Phase) IsAPhase

func (i Phase) IsAPhase() bool

IsAPhase returns "true" if the value is listed in the enum definition. "false" otherwise

func (Phase) IsFailure

func (p Phase) IsFailure() bool

func (Phase) IsSuccess

func (p Phase) IsSuccess() bool

func (Phase) IsTerminal

func (p Phase) IsTerminal() bool

Returns true if the given phase is failure, retryable failure or success

func (Phase) IsWaitingForResources added in v0.5.8

func (p Phase) IsWaitingForResources() bool

func (Phase) String

func (i Phase) String() string

type PhaseInfo

type PhaseInfo struct {
	// contains filtered or unexported fields

Additional info that should be sent to the front end. The Information is sent to the front-end if it meets certain criterion, for example currently, it is sent only if an event was not already sent for

func PhaseInfoFailed

func PhaseInfoFailed(p Phase, err *core.ExecutionError, info *TaskInfo) PhaseInfo

func PhaseInfoFailure

func PhaseInfoFailure(code, reason string, info *TaskInfo) PhaseInfo

func PhaseInfoInitializing

func PhaseInfoInitializing(t time.Time, version uint32, reason string, info *TaskInfo) PhaseInfo

func PhaseInfoNotReady

func PhaseInfoNotReady(t time.Time, version uint32, reason string) PhaseInfo

Return in the case the plugin is not ready to start

func PhaseInfoQueued

func PhaseInfoQueued(t time.Time, version uint32, reason string) PhaseInfo

func PhaseInfoRetryableFailure

func PhaseInfoRetryableFailure(code, reason string, info *TaskInfo) PhaseInfo

func PhaseInfoRunning

func PhaseInfoRunning(version uint32, info *TaskInfo) PhaseInfo

func PhaseInfoSuccess

func PhaseInfoSuccess(info *TaskInfo) PhaseInfo

func PhaseInfoSystemFailure added in v0.3.23

func PhaseInfoSystemFailure(code, reason string, info *TaskInfo) PhaseInfo

func PhaseInfoSystemRetryableFailure added in v0.3.7

func PhaseInfoSystemRetryableFailure(code, reason string, info *TaskInfo) PhaseInfo

func PhaseInfoWaitingForResources

func PhaseInfoWaitingForResources(t time.Time, version uint32, reason string) PhaseInfo

Return in the case the plugin is not ready to start

func (PhaseInfo) Err

func (p PhaseInfo) Err() *core.ExecutionError

func (PhaseInfo) Info

func (p PhaseInfo) Info() *TaskInfo

func (PhaseInfo) Phase

func (p PhaseInfo) Phase() Phase

func (PhaseInfo) Reason

func (p PhaseInfo) Reason() string

func (PhaseInfo) String

func (p PhaseInfo) String() string

func (PhaseInfo) Version

func (p PhaseInfo) Version() uint32

type Plugin

type Plugin interface {
	// Unique ID for the plugin, should be ideally the same the ID in PluginEntry
	GetID() string
	// Properties desired by the plugin from the available set
	GetProperties() PluginProperties
	// The actual method that is invoked for every single task execution. The method should be a non blocking method.
	// It maybe invoked multiple times and hence all actions should be idempotent. If idempotency is not possible look at
	// Transitions to get some system level guarantees
	Handle(ctx context.Context, tCtx TaskExecutionContext) (Transition, error)
	// Called when the task is to be killed/aborted, because the top level entity was aborted or some other failure happened.
	// Abort should always be idempotent
	Abort(ctx context.Context, tCtx TaskExecutionContext) error
	// Finalize is always called, after Handle or Abort. Finalize should be an idempotent operation
	Finalize(ctx context.Context, tCtx TaskExecutionContext) error

Interface for the core Flyte plugin

type PluginEntry

type PluginEntry struct {
	// System wide unique identifier for the plugin
	ID TaskType
	// A list of all the task types for which this plugin is applicable.
	RegisteredTaskTypes []TaskType
	// A Lazy loading function, that will load the plugin. Plugins should be initialized in this method. It is guaranteed
	// that the plugin loader will be called before any Handle/Abort/Finalize functions are invoked
	LoadPlugin PluginLoader
	// Boolean that indicates if this plugin can be used as the default for unknown task types. There can only be
	// one default in the system
	IsDefault bool
	// A list of all task types for which this plugin should be default handler when multiple registered plugins
	// support the same task type. This must be a subset of RegisteredTaskTypes and at most one default per task type
	// is supported.
	DefaultForTaskTypes []TaskType

An entry that identifies the CorePlugin

type PluginLoader

type PluginLoader func(ctx context.Context, iCtx SetupContext) (Plugin, error)

A Lazy loading function, that will load the plugin. Plugins should be initialized in this method. It is guaranteed that the plugin loader will be called before any Handle/Abort/Finalize functions are invoked

type PluginProperties

type PluginProperties struct {
	// Instructs the execution engine to not attempt to cache lookup or write for the node.
	DisableNodeLevelCaching bool

System level properties that this Plugin supports

type PluginStateReader

type PluginStateReader interface {
	// Retrieve state version that is currently stored
	GetStateVersion() uint8
	// Retrieve the typed state in t from the stored value. It also returns the stateversion.
	// If there is no state, t will be zero value, stateversion will be 0
	Get(t interface{}) (stateVersion uint8, err error)

Read previously written plugin state (previous round)

type PluginStateWriter

type PluginStateWriter interface {
	// Only the last call to this method is recorded. All previous calls are overwritten
	// This data is also not accessible until the next round.
	Put(stateVersion uint8, v interface{}) error
	// Resets the state to empty or zero value
	Reset() error

Write new plugin state for a plugin

type ResourceConstraint added in v0.3.9

type ResourceConstraint struct {
	Value int64

type ResourceConstraintsSpec added in v0.3.9

type ResourceConstraintsSpec struct {
	ProjectScopeResourceConstraint   *ResourceConstraint
	NamespaceScopeResourceConstraint *ResourceConstraint

ResourceConstraintsSpec is a contract that a plugin can specify with ResourceManager to force runtime quota-allocation constraints at different levels.

Setting constraints in a ResourceConstraintsSpec to nil objects is valid, meaning there's no constraint at the corresponding level. For example, a ResourceConstraintsSpec with nil ProjectScopeResourceConstraint and a non-nil NamespaceScopeResourceConstraint means that it only poses a cap at the namespace level. A zero-value ResourceConstraintsSpec means there's no constraints posed at any level.

type ResourceManager

type ResourceManager interface {
	GetID() string
	// During execution time, plugins can call AllocateResource() to register a token to the token pool associated with a resource with the resource manager.
	// If granted an allocation, the token will be recorded in the corresponding token pool until the same plugin releases it.
	// When calling AllocateResource, the plugin needs to specify a ResourceConstraintsSpec which contains resource capping constraints at different levels.
	// The ResourceConstraint pointers in ResourceConstraintsSpec, however, can be set to nil to present a non-constraint at that level
	AllocateResource(ctx context.Context, namespace ResourceNamespace, allocationToken string, constraintsSpec ResourceConstraintsSpec) (AllocationStatus, error)
	// During execution time, after an outstanding request is completed, the plugin need to use ReleaseResource() to release the allocation of the corresponding token
	// from the token pool in order to gain back the quota taken by the token
	ReleaseResource(ctx context.Context, namespace ResourceNamespace, allocationToken string) error

ResourceManager Interface 1. Terms and definitions

  • Resource: resource is an abstraction of anything that has a limited quota of units and can be claimed in a single unit or multiple units at once. At Flyte's current state, a resource means a logical separation (e.g., a cluster) of an external service that allows a limited number of outstanding requests to be sent to.
  • Token: Flyte uses a token to serve as the placeholder to represent a unit of resource. Flyte resource manager manages resources by managing the tokens of the resources.
  1. Description ResourceManager provides a task-type-specific pooling system for Flyte Tasks. Plugin writers can optionally request for resources in their tasks, in single quantity.

  2. Usage A Flyte plugin registers the resources and the desired quota of each resource with ResourceRegistrar at the setup time of Flyte Propeller. At the end of the setup time, Flyte Propeller builds a ResourceManager based on these registration requests.

    During runtime, the ResourceManager does two simple things: allocating tokens and releasing tokens. When a Flyte task execution wants to send a request to an external service, the plugin should claim a unit of the corresponding resource. Specifically, an execution needs to generate a unique token, and register the token with ResourceManager by calling ResourceManager's AllocateResource() function. ResourceManager will check its current utilization and the allocation policy to decide whether or not to grant the request. Only when receiving the "AllocationGranted" status shall this execution move forward and send out the request. The granted token will be recorded in a token pool corresponding to the resource and managed by ResourceManager. When the request is done, the plugin will ask the resource manager to release the token by calling ResourceManager's ReleaseResource(), and the token will be erased from the corresponding pool.

  1. Example Flyte has a built-on Qubole plugin that allows Flyte tasks to send out Hive commands to Qubole. In the plugin, a single Qubole cluster is a resource, and sending out a single Hive command to a Qubole cluster consumes a token of the corresponding resource. The resource allocation is achieved by the Qubole plugin calling status, err := AllocateResource(ctx, <cluster name>, <token string>, <constraint spec>) and the de-allocation is achieved by the plugin calling status, err := AllocateResource(ctx, <cluster name>, <token string>, <constraint spec>)

    For example, status, err := AllocateResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0", ResourceConstraintsSpec{}) When the corresponding Hive command finishes, the plugin needs to make the following function call to release the corresponding token err := ReleaseResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0")

type ResourceNamespace

type ResourceNamespace string

func (ResourceNamespace) CreateSubNamespace

func (r ResourceNamespace) CreateSubNamespace(namespace ResourceNamespace) ResourceNamespace

type ResourceRegistrar

type ResourceRegistrar interface {
	RegisterResourceQuota(ctx context.Context, namespace ResourceNamespace, quota int) error

type SecretManager

type SecretManager interface {
	Get(ctx context.Context, key string) (string, error)

type SetupContext

type SetupContext interface {
	// returns a callback mechanism that indicates that (workflow, task) is ready to be re-evaluated
	EnqueueOwner() EnqueueOwner
	// provides a k8s specific owner kind
	OwnerKind() string
	// a metrics scope to publish stats under
	MetricsScope() promutils.Scope
	// A kubernetes client to the bound cluster
	KubeClient() KubeClient
	// Returns a secret manager that can retrieve configured secrets for this plugin
	SecretManager() SecretManager
	// Returns a resource negotiator that the plugin can register resource quota against
	ResourceRegistrar() ResourceRegistrar

Passed to the Loader function when setting up a plugin

type SignalAsync

type SignalAsync func(ctx context.Context)

A simple fire-and-forget func

type TaskExecutionContext

type TaskExecutionContext interface {
	// Returns a resource manager that can be used to create reservations for limited resources
	ResourceManager() ResourceManager

	// Returns a secret manager that can retrieve configured secrets for this plugin
	SecretManager() SecretManager

	// Returns a method that allows a plugin to indicate that the task has a new update and can be invoked again to check for updates
	TaskRefreshIndicator() SignalAsync

	// Returns the max allowed dataset size that the outputwriter will accept
	MaxDatasetSizeBytes() int64

	// Returns a handle to the currently configured storage backend that can be used to communicate with the tasks or write metadata
	DataStore() *storage.DataStore

	// Returns a reader that retrieves previously stored plugin internal state. the state itself is immutable
	PluginStateReader() PluginStateReader

	// Returns a TaskReader, to retrieve task details
	TaskReader() TaskReader

	// Returns an input reader to retrieve input data
	InputReader() io.InputReader

	// Returns a handle to the Task's execution metadata.
	TaskExecutionMetadata() TaskExecutionMetadata

	// Provides an output sync of type io.OutputWriter
	OutputWriter() io.OutputWriter

	// Get a handle to the PluginStateWriter. Any mutations to the plugins internal state can be persisted using this
	// These mutation will be visible in the next round
	PluginStateWriter() PluginStateWriter

	// Get a handle to catalog client
	Catalog() catalog.AsyncClient

	// Returns a handle to the Task events recorder, which get stored in the Admin.
	EventsRecorder() EventsRecorder

An interface that is passed to every plugin invocation. It carries all meta and contextual information for the current task execution

type TaskExecutionID

type TaskExecutionID interface {
	GetGeneratedName() string
	GetID() core.TaskExecutionIdentifier

Simple Interface to expose the ExecutionID of the running Task

type TaskExecutionMetadata

type TaskExecutionMetadata interface {
	// The owning Kubernetes object
	GetOwnerID() types.NamespacedName
	// A specially generated task execution id, that is guaranteed to be unique and consistent for subsequent calls
	GetTaskExecutionID() TaskExecutionID
	GetNamespace() string
	GetOwnerReference() v12.OwnerReference
	GetOverrides() TaskOverrides
	GetLabels() map[string]string
	GetMaxAttempts() uint32
	GetAnnotations() map[string]string
	GetK8sServiceAccount() string
	IsInterruptible() bool

TaskContext represents any execution information for a Task. It is used to communicate meta information about the execution or any previously stored information

type TaskInfo

type TaskInfo struct {
	// log information for the task execution
	Logs []*core.TaskLog
	// Set this value to the intended time when the status occurred at. If not provided, will be defaulted to the current
	// time at the time of publishing the event.
	OccurredAt *time.Time
	// Custom Event information that the plugin would like to expose to the front-end
	CustomInfo *structpb.Struct

func (*TaskInfo) String

func (t *TaskInfo) String() string

type TaskOverrides

type TaskOverrides interface {
	GetResources() *v1.ResourceRequirements
	GetConfig() *v1.ConfigMap

Interface to expose any overrides that have been set for this task (like resource overrides etc)

type TaskReader

type TaskReader interface {
	// Returns the core TaskTemplate
	Read(ctx context.Context) (*core.TaskTemplate, error)

An interface to access the TaskInformation

type TaskType

type TaskType = string

type Transition

type Transition struct {
	// contains filtered or unexported fields

A Plugin Handle method returns a Transition. This transition indicates to the Flyte framework that if the plugin wants to continue "Handle"ing this task, or if wants to move the task to success, attempt a retry or fail. The transition automatically sends an event to Admin service which shows the plugin provided information in the Console/cli etc The information to be published is in the PhaseInfo structure. Transition Type indicates the type of consistency for subsequent handle calls in case the phase info results in a non terminal state. the PhaseInfo structure is very important and is used to record events in Admin. Only if the Phase + PhaseVersion was not previously observed, will an event be published to Admin there are only a configurable number of phase-versions usable. Usually it is preferred to be a monotonically increasing sequence

func DoTransition

func DoTransition(info PhaseInfo) Transition

Same as DoTransition, but TransitionTime is always Ephemeral

func DoTransitionType

func DoTransitionType(ttype TransitionType, info PhaseInfo) Transition

Creates and returns a new Transition based on the PhaseInfo.Phase Phases: PhaseNotReady, PhaseQueued, PhaseInitializing, PhaseRunning will cause the system to continue invoking Handle

func (Transition) Info

func (t Transition) Info() PhaseInfo

func (Transition) String

func (t Transition) String() string
trns := DoTransitionType(TransitionTypeBarrier, PhaseInfoUndefined)

TransitionTypeBarrier,Phase<PhaseUndefined:0 <nil> Reason:>

func (Transition) Type

func (t Transition) Type() TransitionType

type TransitionType

type TransitionType int

Type of Transition, refer to Transition to understand what transition means

const (
	// The transition is eventually consistent. For all the state written may not be visible in the next call, but eventually will persist
	// Best to use when the plugin logic is completely idempotent. This is also the most performant option.
	TransitionTypeEphemeral TransitionType = iota
	// This transition tries its best to make the latest state visible for every consecutive read. But, it is possible
	// to go back in time, i.e. monotonic consistency is violated (in rare cases).

func TransitionTypeString

func TransitionTypeString(s string) (TransitionType, error)

TransitionTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func TransitionTypeValues

func TransitionTypeValues() []TransitionType

TransitionTypeValues returns all values of the enum

func (TransitionType) IsATransitionType

func (i TransitionType) IsATransitionType() bool

IsATransitionType returns "true" if the value is listed in the enum definition. "false" otherwise

func (TransitionType) String

func (i TransitionType) String() string


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL