Kubernetes operator to executes Flyte graphs natively on kubernetes
kubectl-flyte is an command line tool that can be used as an extension to kubectl. It is a separate binary that is built from the propeller repo.
This command will install kubectl-flyte and flytepropeller to
$ make compile
Two ways to execute the command, either standalone kubectl-flyte or as a subcommand of kubectl
$ kubectl-flyte --help OR $ kubectl flyte --help Flyte is a serverless workflow processing platform built for native execution on K8s. It is extensible and flexible to allow adding new operators and comes with many operators built in Usage: kubectl-flyte [flags] kubectl-flyte [command] Available Commands: compile Compile a workflow from core proto-buffer files and output a closure. config Runs various config commands, look at the help of this command to get a list of available commands.. create Creates a new workflow from proto-buffer files. delete delete a workflow get Gets a single workflow or lists all workflows currently in execution help Help about any command visualize Get GraphViz dot-formatted output.
Observing running workflows
To retrieve all workflows in a namespace use the --namespace option, --namespace = "" implies all namespaces.
$ kubectl-flyte get --namespace flytekit-development workflows ├── flytekit-development/flytekit-development-f01c74085110840b8827 [ExecId: ... ] (2m34s Succeeded) - Time SinceCreation(30h1m39.683602s) ... Found 19 workflows Success: 19, Failed: 0, Running: 0, Waiting: 0
To retrieve a specific workflow, namespace can either be provided in the format namespace/name or using the --namespace argument
$ kubectl-flyte get flytekit-development/flytekit-development-ff806e973581f4508bf1 Workflow └── flytekit-development/flytekit-development-ff806e973581f4508bf1 [ExecId: project:"flytekit" domain:"development" name:"ff806e973581f4508bf1" ] (2m32s Succeeded ) ├── start-node start 0s Succeeded ├── c task 0s Succeeded ├── b task 0s Succeeded ├── a task 0s Succeeded └── end-node end 0s Succeeded
To delete a specific workflow
$ kubectl-flyte delete --namespace flytekit-development flytekit-development-ff806e973581f4508bf1
To delete all completed workflows - they have to be either success/failed with a special isCompleted label set on them. The Label is set
$ kubectl-flyte delete --namespace flytekit-development --all-completed
Running propeller locally
use the config.yaml in root found
here <https://github.com/lyft/flytepropeller/blob/master/config.yaml>. Cd into this folder and then run
$ flytepropeller --logtostderr
Following dependencies need to be met
- Blob store (you can forward minio port to localhost)
- Admin Service endpoint (can be forwarded) OR Disable events to admin and launchplans
- access to kubeconfig and kubeapi
Making changes to CRD
Remember changes to CRD should be carefully done, they should be backwards compatible or else you should use proper operator versioning system. Once you do the changes, remember to execute
Package v1alpha1 is the v1alpha1 version of the API.
|Package v1alpha1 is the v1alpha1 version of the API.|
This package has the automatically generated clientset.
|This package has the automatically generated clientset.|
This package has the automatically generated fake clientset.
|This package has the automatically generated fake clientset.|
This package contains the scheme of the automatically generated clientset.
|This package contains the scheme of the automatically generated clientset.|
This package has the automatically generated typed clients.
|This package has the automatically generated typed clients.|
Package fake has the automatically generated clients.
|Package fake has the automatically generated clients.|
This package provides compiler services for flyte workflows.
|This package provides compiler services for flyte workflows.|
This package defines the intermediate layer that the compiler builds and transformers accept.
|This package defines the intermediate layer that the compiler builds and transformers accept.|
This package is a central repository of all compile errors that can be reported.
|This package is a central repository of all compile errors that can be reported.|
This package converts the output of the compiler into a K8s resource for propeller to execute.
|This package converts the output of the compiler into a K8s resource for propeller to execute.|
This package contains validators for all elements of the workflow spec (node, task, branch, interface, bindings...
|This package contains validators for all elements of the workflow spec (node, task, branch, interface, bindings...|