controllerutil

README

Kubernetes Controller Utilities

The github.com/ianlewis/controllerutil package provides a simple way to manage multiple Kubernetes controllers as part of a single process.

Project Status

Project status: alpha

controllerutil is still under active development and has not been extensively tested yet. Use at your own risk. Backward-compatibility is not supported for alpha releases.

Motivation

Kubernetes controllers are often run together as part of a single process but often it's unclear how to manage the lifecycle of each controller. Each controller also makes use of one or more "informers" which are used to watch the Kubernetes API for changes to objects and maintain a local cache per object type. Informers have their own lifecycle and need to managed as well.

This complexity, while powerful, often results in architectures that are error prone and don't handle edge cases well; particularly failure cases. The goal of this library is to provide an easy way for developers of controllers to take advantage of Go programming, logging, and Kubernetes API client best practices.

Installation

Install using

go get github.com/ianlewis/controllerutil

Architecture

controllerutil helps manage client-go data structures. Each controller application watches Kubernetes objects for changes via the Kubernetes API. Watches are done via data structures called "informers". Informer logic is combined with a local cache of objects in a data structure called a "shared index informer. Each shared index informer is shared across the process per API type. Maintaining a cache local to the application is done to lessen the load on the Kubernetes API server.

Each controller application can contain multiple control loops called "controllers". Shared index informers for the necessary types are passed to controllers in their constructor. Each controller is run in parallel with others in a goroutine.

controllerutil implements a ControllerManager which manages each controller and associated goroutine. Since most controller applications require all controllers to properly function, If any controller fails ControllerManager stops all controllers and terminates. It is assumed that the controller will be run with a supervisor such as the kubelet and will be restarted.

controllerutil also has a SharedInformers data structure which manages a list of informers that is unique per Kubernetes API type and manages the associated goroutines.

Usage

This example shows how to use the controllerutil package to implement a Kubernetes controller or operator application that makes use of Kubernetes custom resources. See the example directory for a full example.

Here we create a ControllerManager via the NewControllerManager function. Then controllers are registered via the Register method. Finally we call the Run method to start all the registered controllers.

func Example_customResourceDefinition() {
	// Initialize an in-cluster Kubernetes client
	config, _ := rest.InClusterConfig()
	client, _ := clientset.NewForConfig(config)

	// Create the client for the custom resource definition (CRD) "Foo"
	fooclient, err := fooclientset.NewForConfig(config)
	if err != nil {
		// handle error
	}

	// Create a new ControllerManager instance
	m := controllerutil.NewControllerManager("foo", client)

	// Register the foo controller.
	m.Register("foo", func(ctx *controller.Context) controller.Interface {
		return NewFooController(
			// ctx.Client is the same client passed to controllerutil.New
			ctx.Client,
			// fooclient is the CRD client instance
			fooclient,
			// ctx.SharedInformers manages lifecycle of all shared informers
			// InformerFor registers the informer for the given type if it hasn't been registered already.
			ctx.SharedInformers.InformerFor(
				&examplev1.Foo{},
				func() cache.SharedIndexInformer {
					return fooinformers.NewFooInformer(
						fooclient,
						metav1.NamespaceAll,
						12*time.Hour,
						cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
					)
				},
			),
			// ctx.Recorder is used for recording Kubernetes events.
			ctx.Recorder,
			// ctx.Logger is a convenient wrapper used for logging.
			ctx.Logger,
		)
	})

	if err := m.Run(context.Background()); err != nil {
		// handle error
	}
}

Logging

containerutil provides a logger object per controller which can optionally be used to log messages. Loggers are a wrapper around glog but provides easy creation of standard log.Logger objects. Each logger is given a prefix based on the controller name which allows for easier log viewing.

Disclaimers

This is not an official Google product.

Documentation

Overview

Package controllerutil implements a ControllerManager type that is used to manage and run several Kubernetes controllers as part of a single process.

Controller are registered with the ControllerManager via the Register method and the group of controllers are run using the Run method on the ControllerManager.

Example (CustomResourceDefinition)

package main

import (
	"context"
	"time"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	clientset "k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/cache"
	"k8s.io/client-go/tools/record"

	"github.com/ianlewis/controllerutil"
	"github.com/ianlewis/controllerutil/controller"
	"github.com/ianlewis/controllerutil/logging"

	// CRD type definitions
	examplev1 "github.com/ianlewis/controllerutil/example/pkg/apis/example.com/v1"
	// Generated CRD client
	fooclientset "github.com/ianlewis/controllerutil/example/pkg/client/clientset/versioned"
	// Generated CRD informer
	fooinformers "github.com/ianlewis/controllerutil/example/pkg/client/informers/externalversions/example/v1"
)

// FooController is a controller that watches Foo custom resource instances.
type FooController struct {
	client      clientset.Interface
	fooClient   fooclientset.Interface
	fooInformer cache.SharedIndexInformer
	recorder    record.EventRecorder
	l           *logging.Logger
}

func (c *FooController) Run(ctx context.Context) error {
	// run control loop
	return nil
}

// NewFooController creates a new controller instance
func NewFooController(
	client clientset.Interface,
	fooClient fooclientset.Interface,
	fooInformer cache.SharedIndexInformer,
	recorder record.EventRecorder,
	l *logging.Logger,
) *FooController {
	c := &FooController{
		client:      client,
		fooClient:   fooClient,
		fooInformer: fooInformer,
		recorder:    recorder,
		l:           l,
	}

	// Attach event handlers
	// fooInformer.AddEventHandler(...)

	return c
}

func main() {
	// Initialize an in-cluster Kubernetes client
	config, _ := rest.InClusterConfig()
	client, _ := clientset.NewForConfig(config)

	// Create the client for the custom resource definition (CRD) "Foo"
	fooclient, err := fooclientset.NewForConfig(config)
	if err != nil {
		// handle error
	}

	// Create a new ControllerManager instance
	m := controllerutil.NewControllerManager("foo", client)

	// Register the foo controller.
	m.Register("foo", func(ctx *controller.Context) controller.Interface {
		return NewFooController(
			// ctx.Client is the same client passed to controllerutil.New
			ctx.Client,
			// fooclient is the CRD client instance
			fooclient,
			// ctx.SharedInformers manages lifecycle of all shared informers
			// InformerFor registers the informer for the given type if it hasn't been registered already.
			ctx.SharedInformers.InformerFor(
				&examplev1.Foo{},
				func() cache.SharedIndexInformer {
					return fooinformers.NewFooInformer(
						fooclient,
						metav1.NamespaceAll,
						12*time.Hour,
						cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
					)
				},
			),
			// ctx.Recorder is used for recording Kubernetes events.
			ctx.Recorder,
			// ctx.Logger is a convenient wrapper used for logging.
			ctx.Logger,
		)
	})

	if err := m.Run(context.Background()); err != nil {
		// handle error
	}
}

Index

• type ControllerManager
  • func NewControllerManager(name string, client clientset.Interface) *ControllerManager
  • func (m *ControllerManager) Register(name string, c controller.Constructor)
  • func (m *ControllerManager) Run(ctx context.Context) error

Examples

• Package (CustomResourceDefinition)
• ControllerManager
• ControllerManager.Register

Types

type ControllerManager

type ControllerManager struct {
	// contains filtered or unexported fields
}

ControllerManager manages a set of controllers registered to it. It manages their lifecycle and the lifecycle of informers that controllers use.

Example

package main

import (
	"context"

	clientset "k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"

	"github.com/ianlewis/controllerutil"
)

func main() {
	config, _ := rest.InClusterConfig()
	client, _ := clientset.NewForConfig(config)
	m := controllerutil.NewControllerManager("hoge", client)

	err := m.Run(context.Background())
	if err != nil {
		// handle error
	}
}

func NewControllerManager

func NewControllerManager(name string, client clientset.Interface) *ControllerManager

New creates a new controller manager.

func (*ControllerManager) Register

func (m *ControllerManager) Register(name string, c controller.Constructor)

Register registers a controller created by the given constructor by the given name. The given name should be unique to the controller and is used in Kubernetes event recorders and logging.

Example

package main

import (
	"context"

	clientset "k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/record"

	"github.com/ianlewis/controllerutil"
	"github.com/ianlewis/controllerutil/controller"
	"github.com/ianlewis/controllerutil/logging"
)

type HogeController struct {
	client   clientset.Interface
	recorder record.EventRecorder
	l        *logging.Logger
}

func (c *HogeController) Run(ctx context.Context) error { return nil }

func main() {
	config, _ := rest.InClusterConfig()
	client, _ := clientset.NewForConfig(config)
	m := controllerutil.NewControllerManager("hoge", client)

	// Register the hoge controller. ctx is a controller context.
	m.Register("hoge", func(ctx *controller.Context) controller.Interface {
		return &HogeController{
			// ctx.Client is the same client passed to controllerutil.New
			client: ctx.Client,
			// ctx.Recorder is used for recording Kubernetes events.
			recorder: ctx.Recorder,
			// ctx.InfoLogger is a convenient wrapper used for logging.
			l: ctx.Logger,
		}
	})
}

func (*ControllerManager) Run

func (m *ControllerManager) Run(ctx context.Context) error

Run starts all controllers and informers registered to the ControllerManager instance. ControllerManager assumes all registered controllers are essential to proper functioning. ControllerManager cancels all controllers and returns the first error returned by the controllers in the event of a failure. ControllerManager will not attempt to restart controllers and will simply return. As a best practice the calling process should log the returned error message and exit. It is assumed that the controller manager will be run by a supervisor process like supervisord or the Kubernetes kubelet and will be restarted gracefully if fatal errors occur.

The ControllerManager starts shared informers and waits for their caches to be synced before starting controllers. Controllers do not need to wait for informers to sync.