krt

package
v0.0.0-...-91868f7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 1, 2024 License: Apache-2.0 Imports: 25 Imported by: 0

README

krt: Kubernetes Declarative Controller Runtime

krt provides a framework for building declarative controllers. See the design doc and KubeCon talk for more background.

The framework aims to solve a few problems with writing controllers:

  • Operate on any types, from any source. Kubernetes provides informers, but these only work on Kubernetes types read from Kubernetes objects.
    • krt can accept any object type, from any source, and handle them the same.
  • Provide high level abstractions.
    • Controller authors can write simple transformation functions from Input -> Output (with dependencies); the framework handles all the state automatically.

Key Primitives

The most important primitive provided is the Collection interface. This is basically an Informer, but not tied to Kubernetes.

Currently, there are three ways to build a Collection:

  • Built from an Informer with WrapClient or NewInformer.
  • Statically configured with NewStatic.
  • Derived from other collections (more information on this below).

Unlike Informers, these primitives work on arbitrary objects. However, these objects are expected to have some properties, depending on their usage. These are not expressed as generic constraints due to limitations in Go's type system.

  • Each object T must have a unique Key[T] (which is just a typed wrapper around string) that uniquely identifies the object. Default implementations exist for Kubernetes objects, Istio config.Config objects, and ResourceName() string implementations.
  • Equals(k K) bool may be implemented to provide custom implementations to compare objects. Comparison is done to detect if changes were made. Default implementations are available for Kubernetes and protobuf objects, and will fallback to reflect.DeepEqual.
  • A Name, Namespace, Labels, and LabelSelector may optionally be included, for use with filters (see below).

Derived Collections

The core of the framework is in the ability to derive collections from others.

In general, these are built by providing some func(inputs...) outputs... (called "transformation" functions). While more could be expressed, there are currently three forms implemented.

  • func() *O via NewSingleton
    • This generates a collection that has a single value. An example would be some global configuration.
  • func(input I) *O via NewCollection
    • This generates a one-to-one mapping of input to output. An example would be a transformation from a Pod type to a generic Workload type.
  • func(input I) []O via NewManyCollection
    • This generates a one-to-many mapping of input to output. An example would be a transformation from a Service to a set of Endpoint types.
    • The order of the response does not matter. Each response must have a unique key.

The form used and input type only represent the primary dependencies, indicating the cardinality. Each transformation can additionally include an arbitrary number of dependencies, fetching data from other collections.

For example, a simple Singleton example that keeps track of the number of ConfigMaps in the cluster:

ConfigMapCount := krt.NewSingleton[int](func(ctx krt.HandlerContext) *int {
    cms := krt.Fetch(ctx, ConfigMaps)
    return ptr.Of(len(cms))
})

The Fetch operation enables querying against other collections. If the result of the Fetch operation changes, the collection will automatically be recomputed; the framework handles the state and event detection. In the above example, the provided function will be called (at least) every time there is a change to a configmap. The ConfigMapCount collection will produce events only when the count changes. The framework will use generic Equals on the underlying object to determine whether or not to recompute collections.

Picking a collection type

There are a variety of collection types available. Picking these is about simplicity, usability, and performance.

The NewSingleton form (func() *O), in theory, could be used universally. Consider a transformation from Pod to SimplePod:

SimplePods := krt.NewSingleton[SimplePod](func(ctx krt.HandlerContext) *[]SimplePod {
    res := []SimplePod{}
    for _, pod := range krt.Fetch(ctx, Pod) {
        res = append(res, SimplePod{Name: pod.Name})
    }
    return &res
}) // Results in a Collection[[]SimplePod]

While this works, it is inefficient and complex to write. Consumers of SimplePod can only query the entire list at once. Anytime any Pod changes, all SimplePods must be recomputed.

A better approach would be to lift Pod into a primary dependency:

SimplePods := krt.NewCollection[SimplePod](func(ctx krt.HandlerContext, pod *v1.Pod) *SimplePod {
    return &SimplePod{Name: pod.Name}
}) // Results in a Collection[SimplePod]

Not only is this simpler to write, its far more efficient. Consumers can more efficiently query for SimplePods using label selectors, filters, etc. Additionally, if a single Pod changes we only recompute one SimplePod.

Above we have a one-to-one mapping of input and output. We may have one-to-many mappings, though. In these cases, usually its best to use a ManyCollection. Like the above examples, its possible to express these as normal Collections, but likely inefficient.

Example computing a list of all container names across all pods:

ContainerNames := krt.NewManyCollection[string](func(ctx krt.HandlerContext, pod *v1.Pod) (res []string) {
    for _, c := range pod.Spec.Containers {
      res = append(res, c.Name)
    }
    return res
}) // Results in a Collection[string]

Example computing a list of service endpoints, similar to the Kubernetes core endpoints controller:

Endpoints := krt.NewManyCollection[Endpoint](func(ctx krt.HandlerContext, svc *v1.Service) (res []Endpoint) {
    for _, c := range krt.Fetch(ctx, Pods, krt.FilterLabel(svc.Spec.Selector)) {
      res = append(res, Endpoint{Service: svc.Name, Pod: pod.Name, IP: pod.status.PodIP})
    }
    return res
}) // Results in a Collection[Endpoint]

As a rule of thumb, if your Collection type is a list, you most likely should be using a different type to flatten the list. An exception to this would be if the list represents an atomic set of items that are never queried independently; in these cases, however, it is probably best to wrap it in a struct. For example, to represent the set of containers in a pod, we may make a type PodContainers struct { Name string, Containers []string } and have a Collection[PodContainers] rather than a Collection[[]string].

In theory, other forms could be expressed such as func(input1 I1, input2 I2) *O. However, there haven't yet been use cases for these more complex forms.

Transformation constraints

In order for the framework to properly handle dependencies and events, transformation functions must adhere by a few properties.

Basically, Transformations must be stateless and idempotent.

  • Any querying of other Collections must be done through krt.Fetch.
  • Querying other data stores that may change is not permitted.
  • Querying external state (e.g. making HTTP calls) is not permitted.
  • Transformations may be called at any time, including many times for the same inputs. Transformation functions should not make any assumptions about calling patterns.

Violation of these properties will result in undefined behavior (which would likely manifest as stale data).

Fetch details

In addition to simply fetching all resources from a collection, a filter can be provided. This is more efficient than filtering outside of Fetch, as the framework can filter un-matched objects earlier, skipping redundant work. The following filters are provided

  • FilterName(name, namespace): filters an object by Name and Namespace.
  • FilterNamespace(namespace): filters an object by Namespace.
  • FilterKey(key): filters an object by key.
  • FilterLabel(labels): filters to only objects that match these labels.
  • FilterSelects(labels): filters to only objects that select these labels. An empty selector matches everything.
  • FilterSelectsNonEmpty(labels): filters to only objects that select these labels. An empty selector matches nothing.
  • FilterGeneric(func(any) bool): filters by an arbitrary function.

Note that most filters may only be used if the objects being Fetched implement appropriate functions to extract the fields filtered against. Failures to meet this requirement will result in a panic.

Library Status

This library is currently "experimental" and is not used in Istio production yet. The intent is this will be slowly rolled out to controllers that will benefit from it and are lower risk; likely, the ambient controller will be the first target.

While its plausible all of Istio could be fundamentally re-architected to fully embrace krt throughout (replacing things like PushContext), it is not yet clear this is desired.

Performance

Compared to a perfectly optimized hand-written controller, krt adds some overhead. However, writing a perfectly optimized controller is hard, and often not done. As a result, for many scenarios it is expected that krt will perform on-par or better.

This is similar to a comparison between a high level programming language compared to assembly; while its always possible to write better code in assembly, smart compilers can make optimizations humans are unlikely to, such as loop unrolling. Similarly, krt can make complex optimizations in one place, so each controller implementation doesn't, which is likely to increase the amount of optimizations applied.

The BenchmarkControllers puts this to the test, comparing an ideal hand-written controller to one written in krt. While the numbers are likely to change over time, at the time of writing the overhead for krt is roughly 10%:

name                  time/op
Controllers/krt-8     13.4ms ±23%
Controllers/legacy-8  11.4ms ± 6%

name                  alloc/op
Controllers/krt-8     15.2MB ± 0%
Controllers/legacy-8  12.9MB ± 0%
Future work
Object optimizations

One important aspect of krt is its ability to automatically detect if objects have changed, and only trigger dependencies if so. This works better when we only compare fields we actually use. Today, users can do this manually by making a transformation from the full object to a subset of the object.

This could be improved by:

  • Automagically detecting which subset of the object is used, and optimize this behind the scenes. This seems unrealistic, though.
  • Allow a lightweight form of a Full -> Subset transformation, that doesn't create a full new collection (with respective overhead), but rather overlays on top of an existing one.
Internal dependency optimizations

Today, the library stores a mapping of Input -> Dependencies (map[Key[I]][]dependency). Often times, there are common dependencies amongst keys. For example, a namespace filter probably has many less unique values than unique input objects. Other filters may be completely static and shared by all keys.

This could be improved by:

  • Optimize the data structure to be a bit more advanced in sharing dependencies between keys.
  • Push the problem to the user; allow them to explicitly set up static Fetches.
Debug tooling

krt has an opportunity to add a lot of debugging capabilities that are hard to do elsewhere, because it would require linking up disparate controllers, and a lot of per-controller logic.

Some debugging tooling ideas:

  • Add OpenTelemetry tracing to controllers (prototype).
  • Automatically generate mermaid diagrams showing system dependencies.
  • Automatically detect violations of Transformation constraints.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BatchedEventFilter 

func BatchedEventFilter[I, O any](cf func(a I) O, handler func(events []Event[I], initialSync bool)) func(o []Event[I], initialSync bool)

BatchedEventFilter allows an event handler to have alternative event suppression mechanics to filter out unnecessary events. For instance, I can make a transformation from `object => object.name` to only trigger events for changes to the name; the output will be compared (using standard equality checking), and only changes will trigger the handler. Note this is in addition to the normal event mechanics, so this can only filter things further.

func Dump 

func Dump[O any](c Collection[O])

Dump is a *testing* helper to dump the state of a collection, if possible, into logs.

func Fetch 

func Fetch[T any](ctx HandlerContext, cc Collection[T], opts ...FetchOption) []T

func FetchOne 

func FetchOne[T any](ctx HandlerContext, c Collection[T], opts ...FetchOption) *T

Types

type Collection 

type Collection[T any] interface {
	// GetKey returns an object by its key, if present. Otherwise, nil is returned.
	GetKey(k Key[T]) *T

	// List returns all objects in the collection.
	// Order of the list is undefined.
	List() []T

	EventStream[T]
}

Collection is the core resource type for krt, representing a collection of objects. Items can be listed, or fetched directly. Most importantly, consumers can subscribe to events when objects change.

func JoinCollection 

func JoinCollection[T any](cs []Collection[T], opts ...CollectionOption) Collection[T]

JoinCollection combines multiple Collection[T] into a single Collection[T] merging equal objects into one record in the resulting Collection

func NewCollection 

func NewCollection[I, O any](c Collection[I], hf TransformationSingle[I, O], opts ...CollectionOption) Collection[O]

NewCollection transforms a Collection[I] to a Collection[O] by applying the provided transformation function. This applies for one-to-one relationships between I and O. For zero-to-one, use NewSingleton. For one-to-many, use NewManyCollection.

func NewInformer 

func NewInformer[I controllers.ComparableObject](c kube.Client, opts ...CollectionOption) Collection[I]

NewInformer creates a Collection[I] sourced from the results of kube.Client querying resources of type I from the API Server.

Resources must have their GVR and GVK registered in the kube.Client before this method is called, otherwise NewInformer will panic.

func NewInformerFiltered 

func NewInformerFiltered[I controllers.ComparableObject](c kube.Client, filter kubetypes.Filter, opts ...CollectionOption) Collection[I]

NewInformerFiltered takes an argument that filters the results from the kube.Client. Otherwise, behaves the same as NewInformer

func NewManyCollection 

func NewManyCollection[I, O any](c Collection[I], hf TransformationMulti[I, O], opts ...CollectionOption) Collection[O]

NewManyCollection transforms a Collection[I] to a Collection[O] by applying the provided transformation function. This applies for one-to-many relationships between I and O. For zero-to-one, use NewSingleton. For one-to-one, use NewCollection.

func NewStaticCollection 

func NewStaticCollection[T any](vals []T) Collection[T]

func WrapClient 

func WrapClient[I controllers.ComparableObject](c kclient.Informer[I], opts ...CollectionOption) Collection[I]

WrapClient is the base entrypoint that enables the creation of a collection from an API Server client.

Generic types can use kclient.NewDynamic to create an informer for a Collection of type controllers.Object

type CollectionOption 

type CollectionOption func(*collectionOptions)

CollectionOption is a functional argument type that can be passed to Collection constructors.

func WithName 

func WithName(name string) CollectionOption

WithName allows explicitly naming a controller. This is a best practice to make debugging easier. If not set, a default name is picked.

func WithObjectAugmentation 

func WithObjectAugmentation(fn func(o any) any) CollectionOption

WithObjectAugmentation allows transforming an object into another for usage throughout the library. Currently this applies to things like Name, Namespace, Labels, LabelSelector, etc. Equals is not currently supported, but likely in the future. The intended usage is to add support for these fields to collections of types that do not implement the appropriate interfaces. The conversion function can convert to a embedded struct with extra methods added: 

type Wrapper struct { Object }
func (w Wrapper) ResourceName() string { return ... }
WithObjectAugmentation(func(o any) any { return Wrapper{o.(Object)} })

func WithStop 

func WithStop(stop <-chan struct{}) CollectionOption

WithStop sets a custom stop channel so a collection can be terminated when the channel is closed

type Equaler 

type Equaler[K any] interface {
	Equals(k K) bool
}

Equaler is an optional interface that can be implemented by collection types. If implemented, this will be used to determine if an object changed.

type Event 

type Event[T any] struct {
	// Old object, set on Update or Delete.
	Old *T
	// New object, set on Add or Update
	New *T
	// Event is the change type
	Event controllers.EventType
}

Event represents a point in time change for a collection.

func (Event[T]) Items 

func (e Event[T]) Items() []T

Items returns both the Old and New object, if present.

func (Event[T]) Latest 

func (e Event[T]) Latest() T

Latest returns only the latest object (New for add/update, Old for delete).

type EventStream 

type EventStream[T any] interface {
	// Register adds an event watcher to the collection. Any time an item in the collection changes, the handler will be
	// called. Typically, usage of Register is done internally in krt via composition of Collections with Transformations
	// (NewCollection, NewManyCollection, NewSingleton); however, at boundaries of the system (connecting to something not
	// using krt), registering directly is expected.
	Register(f func(o Event[T])) Syncer

	// Synced returns a Syncer which can be used to determine if the collection has synced. Once its synced, all dependencies have
	// been processed, and all handlers have been called with the results.
	Synced() Syncer

	// RegisterBatch registers a handler that accepts multiple events at once. This can be useful as an optimization.
	// Otherwise, behaves the same as Register.
	// Additionally, skipping the default behavior of "send all current state through the handler" can be turned off.
	// This is important when we register in a handler itself, which would cause duplicative events.
	RegisterBatch(f func(o []Event[T], initialSync bool), runExistingState bool) Syncer
}

EventStream provides a link between the underlying collection and its clients. The EventStream does not publish events for retrigger operations where the resultant object of type T is equal to an existing object in the collection.

On initial sync, events will be published to registered clients as the Collection is populated.

type FetchOption 

type FetchOption func(*dependency)

FetchOption is a functional argument type that can be passed to Fetch. These are all created by the various Filter* functions

func FilterGeneric 

func FilterGeneric(f func(any) bool) FetchOption

func FilterIndex 

func FilterIndex[I any, K comparable](idx *Index[I, K], k K) FetchOption

FilterIndex selects only objects matching a key in an index.

func FilterKey 

func FilterKey(k string) FetchOption

func FilterKeys 

func FilterKeys(k ...string) FetchOption

func FilterLabel 

func FilterLabel(lbls map[string]string) FetchOption

func FilterObjectName 

func FilterObjectName(name types.NamespacedName) FetchOption

FilterObjectName selects a Kubernetes object by name.

func FilterSelects 

func FilterSelects(lbls map[string]string) FetchOption

FilterSelects only includes objects that select this label. If the selector is empty, it is a match.

func FilterSelectsNonEmpty 

func FilterSelectsNonEmpty(lbls map[string]string) FetchOption

FilterSelectsNonEmpty only includes objects that select this label. If the selector is empty, it is not a match.

type HandlerContext 

type HandlerContext interface {
	// contains filtered or unexported methods
}

HandlerContext is an opaque type passed into transformation functions. This can be used with Fetch to dynamically query for resources. Note: this doesn't expose Fetch as a method, as Go generics do not support arbitrary generic types on methods.

type Index 

type Index[I any, K comparable] struct {
	// contains filtered or unexported fields
}

Index maintains a simple index over an informer

func NewIndex 

func NewIndex[I any, K comparable](
	c Collection[I],
	extract func(o I) []K,
) *Index[I, K]

NewIndex creates a simple index, keyed by key K, over an informer for O. This is similar to NewInformer.AddIndex, but is easier to use and can be added after an informer has already started.

func NewNamespaceIndex 

func NewNamespaceIndex[I Namespacer](c Collection[I]) *Index[I, string]

NewNamespaceIndex is a small helper to index a collection by namespace

func (*Index[I, K]) Dump 

func (i *Index[I, K]) Dump()

func (*Index[I, K]) Lookup 

func (i *Index[I, K]) Lookup(k K) []I

Lookup finds all objects matching a given key

type Key 

type Key[O any] string

Key is a string, but with a type associated to avoid mixing up keys

func GetApplyConfigKey 

func GetApplyConfigKey[O any](a O) *Key[O]

GetApplyConfigKey returns the key for the ApplyConfig. If there is none, this will return nil.

func GetKey 

func GetKey[O any](a O) Key[O]

GetKey returns the key for the provided object. If there is none, this will panic.

type LabelSelectorer 

type LabelSelectorer interface {
	GetLabelSelector() map[string]string
}

LabelSelectorer is an optional interface that can be implemented by collection types. If implemented, this will be used to determine an objects' LabelSelectors

type Labeler 

type Labeler interface {
	GetLabels() map[string]string
}

Labeler is an optional interface that can be implemented by collection types. If implemented, this will be used to determine an objects' Labels

type Named 

type Named struct {
	Name, Namespace string
}

Named is a convenience struct. It is ideal to be embedded into a type that has a name and namespace, and will automatically implement the various interfaces to return the name, namespace, and a key based on these two.

func NewNamed 

func NewNamed(o metav1.Object) Named

NewNamed builds a Named object from a Kubernetes object type.

func (Named) GetName 

func (n Named) GetName() string

func (Named) GetNamespace 

func (n Named) GetNamespace() string

func (Named) ResourceName 

func (n Named) ResourceName() string

type Namer 

type Namer interface {
	GetName() string
}

Namer is an optional interface that can be implemented by collection types. If implemented, this will be used to determine an objects' Name.

type Namespacer 

type Namespacer interface {
	GetNamespace() string
}

Namespacer is an optional interface that can be implemented by collection types. If implemented, this will be used to determine an objects' Namespace.

type RecomputeTrigger 

type RecomputeTrigger struct {
	// contains filtered or unexported fields
}

RecomputeTrigger trigger provides an escape hatch to allow krt transformations to depend on external state and recompute correctly when those change. Typically, all state is registered and fetched through krt.Fetch. Through this mechanism, any changes are automatically propagated through the system to dependencies. In some cases, it may not be feasible to get all state into krt; hopefully, this is a temporary state. RecomputeTrigger works around this by allowing an explicit call to recompute a collection; the caller must be sure to call Trigger() any time the state changes.

func NewRecomputeTrigger 

func NewRecomputeTrigger() *RecomputeTrigger

func (*RecomputeTrigger) MarkDependant 

func (r *RecomputeTrigger) MarkDependant(ctx HandlerContext)

MarkDependant marks the given context as depending on this trigger. This registers it to be recomputed when TriggerRecomputation is called.

func (*RecomputeTrigger) TriggerRecomputation 

func (r *RecomputeTrigger) TriggerRecomputation()

TriggerRecomputation tells all dependants to recompute

type ResourceNamer 

type ResourceNamer interface {
	ResourceName() string
}

ResourceNamer is an optional interface that can be implemented by collection types. If implemented, this can be used to determine the Key for an object

type Singleton 

type Singleton[T any] interface {
	// Get returns the object, or nil if there is none.
	Get() *T
	// Register adds an event watcher to the object. Any time it changes, the handler will be called
	Register(f func(o Event[T])) Syncer
	AsCollection() Collection[T]
}

Singleton is a special Collection that only ever has a single object. They can be converted to the Collection where convenient, but when using directly offer a more ergonomic API

func NewSingleton 

func NewSingleton[O any](hf TransformationEmpty[O], opts ...CollectionOption) Singleton[O]

type StaticSingleton 

type StaticSingleton[T any] interface {
	Singleton[T]
	Set(*T)
}

func NewStatic 

func NewStatic[T any](initial *T) StaticSingleton[T]

type Syncer 

type Syncer interface {
	WaitUntilSynced(stop <-chan struct{}) bool
	HasSynced() bool
}

type TestingDummyContext 

type TestingDummyContext struct{}

type TransformationEmpty 

type TransformationEmpty[T any] func(ctx HandlerContext) *T

TransformationEmpty represents a singleton operation. There is always a single output. Note this can still depend on other types, via Fetch.

type TransformationMulti 

type TransformationMulti[I, O any] func(ctx HandlerContext, i I) []O

TransformationMulti represents a one-to-many relationship between I and O.

type TransformationSingle 

type TransformationSingle[I, O any] func(ctx HandlerContext, i I) *O

TransformationSingle represents a one-to-one relationship between I and O.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.