motion

Documentation

Index

• func SetupMotionControllers(mgr manager.Manager) error
• func VolumeConfiguration(batchTransfer *motionv1.BatchTransfer) ([]v1.Volume, []v1.VolumeMount, error)
• type BatchTransferReconciler
  • func NewBatchTransferReconciler(mgr ctrl.Manager, name string) *BatchTransferReconciler
  • func (reconciler *BatchTransferReconciler) CreateBatchJob(batchTransfer *motionv1.BatchTransfer) error
  • func (reconciler *BatchTransferReconciler) CreateSecret(batchTransfer *motionv1.BatchTransfer) error
  • func (reconciler *BatchTransferReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
  • func (reconciler *BatchTransferReconciler) SetupWithManager(mgr ctrl.Manager) error
• type Reconciler
• type StreamTransferReconciler
  • func NewStreamTransferReconciler(mgr ctrl.Manager, name string) *StreamTransferReconciler
  • func (reconciler *StreamTransferReconciler) CreateDeployment(streamTransfer *motionv1.StreamTransfer) error
  • func (reconciler *StreamTransferReconciler) CreatePVC(streamTransfer *motionv1.StreamTransfer) error
  • func (reconciler *StreamTransferReconciler) CreateSecret(streamTransfer *motionv1.StreamTransfer) error
  • func (reconciler *StreamTransferReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
  • func (reconciler *StreamTransferReconciler) SetupWithManager(mgr ctrl.Manager) error

Functions

func SetupMotionControllers

func SetupMotionControllers(mgr manager.Manager) error

This function sets up all motion controllers including the webhooks given a controller manager. Webhooks can be activated/deactivated using the ENABLE_WEBHOOKS environment variable. This currently includes: - a manager for BatchTransfers - a manager for StreamTransfers

func VolumeConfiguration

func VolumeConfiguration(batchTransfer *motionv1.BatchTransfer) ([]v1.Volume, []v1.VolumeMount, error)

Returns the volume configuration for a BatchTransfer. This includes the configuration secret as well as possible secrets that have to be mounted that contain trust stores and additional secrets that are not defined in the configuration secret. TODO: Make this usable with the motionv1.Transfer so that it works with different as well

Types

type BatchTransferReconciler

type BatchTransferReconciler struct {
	Reconciler
}

BatchTransferReconciler reconciles a BatchTransfer object It is "derived" from the Reconciler object

func NewBatchTransferReconciler

func NewBatchTransferReconciler(mgr ctrl.Manager, name string) *BatchTransferReconciler

NewBatchTransferReconciler creates a new reconciler for BatchTransfer resources

func (*BatchTransferReconciler) CreateBatchJob

func (reconciler *BatchTransferReconciler) CreateBatchJob(batchTransfer *motionv1.BatchTransfer) error

Create one of two jobs: A spark job for tabular data or a binary copy job.

func (*BatchTransferReconciler) CreateSecret

func (reconciler *BatchTransferReconciler) CreateSecret(batchTransfer *motionv1.BatchTransfer) error

Create the secret that is used for the BatchTransfer object This secret contains the spec of the BatchTransfer object in a JSON file.

func (*BatchTransferReconciler) Reconcile

func (reconciler *BatchTransferReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

This is the main entry point of the controller. It reconciles BatchTransfer objects. The batch transfer is implemented as a K8s Job or CronJob. Reconciliation happens with the following steps: - Fetch the BatchTransfer object - Check if the object is being deleted and handle a finalizer if needed - Update the status by checking the existing Job/CronJob - If K8s objects are not yet created create the objects

func (*BatchTransferReconciler) SetupWithManager

func (reconciler *BatchTransferReconciler) SetupWithManager(mgr ctrl.Manager) error

Setup the reconciler. This consists of creating an index of jobs where this controller is the owner.

type Reconciler

type Reconciler struct {
	client.Client
	Name   string
	Log    logr.Logger
	Scheme *runtime.Scheme
}

A reconciler can be used to base reconcilers of Transfers on. It has functions on how to manage finalizers for transfers and other utilities that can be used by specific reconcilers. It is "derived" (using type embedding) from the K8s client

type StreamTransferReconciler

type StreamTransferReconciler struct {
	Reconciler
}

StreamTransferReconciler reconciles a StreamTransfer object It is "derived" from the Reconciler object

func NewStreamTransferReconciler

func NewStreamTransferReconciler(mgr ctrl.Manager, name string) *StreamTransferReconciler

NewStreamTransferReconciler creates a new reconciler for StreamTransfer resources

func (*StreamTransferReconciler) CreateDeployment

func (reconciler *StreamTransferReconciler) CreateDeployment(streamTransfer *motionv1.StreamTransfer) error

Create the deployment for a stream transfer object. This deployment will mount the PVC for the checkpoint directory as well as the secret for the configuration.

func (*StreamTransferReconciler) CreatePVC

func (reconciler *StreamTransferReconciler) CreatePVC(streamTransfer *motionv1.StreamTransfer) error

Create the persistent volume claim for the checkpoint directory This PVC stores the state of a given transfer so that it can be restarted from a given checkpoint in case of failure.

func (*StreamTransferReconciler) CreateSecret

func (reconciler *StreamTransferReconciler) CreateSecret(streamTransfer *motionv1.StreamTransfer) error

Create the secret that is used for the StreamTransfer object This secret contains the spec of the StreamTransfer object in a JSON file.

func (*StreamTransferReconciler) Reconcile

func (reconciler *StreamTransferReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile StreamTransfers A first version of this StreamTransfer is based on K8s Deployment objects. These manage pod failures themselves and restart them in case of failure. Thus in this first version that does not handle errors yet a stream will be started if it does not exist (including a persistent checkpoint storage) and otherwise be left running. A more involved version that is handling errors and is discovering crash loops may have to be based on a Pod directly in order to discover errors on a finer granular level.

func (*StreamTransferReconciler) SetupWithManager

func (reconciler *StreamTransferReconciler) SetupWithManager(mgr ctrl.Manager) error