dra

README

Overview

The tests in this directory cover dynamic resource allocation support in Kubernetes. They do not test the correct behavior of arbitrary dynamic resource allocation drivers.

If such a driver is needed, then the in-tree test/e2e/dra/test-driver is used, with a slight twist: instead of deploying that driver directly in the cluster, the necessary sockets for interaction with kubelet (registration and dynamic resource allocation) get proxied into the e2e.test binary. This reuses the work done for CSI mock testing. The advantage is that no separate images are needed for the test driver and that the e2e test has full control over all gRPC calls, in case that it needs that for operations like error injection or checking calls.

Cluster setup preparation

The container runtime must support CDI. CRI-O supports CDI starting from release 1.23, Containerd supports CDI starting from release 1.7. To bring up a Kind cluster with Containerd, two things are needed:

• build binaries from Kubernetes source code tree
• Kind

NB: Kind switched to use worker-node base image with Containerd 1.7 by default starting from release 0.20, build kind from latest main branch sources or use Kind release binary 0.20 or later.

Build kind node image

After building Kubernetes, in Kubernetes source code tree build new node image:

$ kind build node-image --image dra/node:latest $(pwd)

Bring up a Kind cluster

$ kind create cluster --config test/e2e/dra/kind.yaml --image dra/node:latest

Run tests

• Build ginkgo

$ make ginkgo

• Run e2e tests for the Dynamic Resource Allocation feature:

$ KUBECONFIG=~/.kube/config _output/bin/ginkgo -p -v -focus=Feature:DynamicResourceAllocation ./test/e2e

Documentation

Index

• Constants
• type Driver
  • func NewDriver(f *framework.Framework, nodes *Nodes, configureResources func() Resources, ...) *Driver
  • func NewDriverInstance(f *framework.Framework) *Driver
  • func (d *Driver) CallCount(m MethodInstance) int64
  • func (d *Driver) Fail(m MethodInstance, injectError bool)
  • func (d *Driver) IsGone(ctx context.Context)
  • func (d *Driver) NewGetSlices() framework.GetFunc[*resourceapi.ResourceSliceList]
  • func (d *Driver) Nodenames() (nodenames []string)
  • func (d *Driver) Run(nodes *Nodes, configureResources func() Resources, ...)
  • func (d *Driver) SetUp(nodes *Nodes, resources Resources, ...)
  • func (d *Driver) TearDown(ctx context.Context)
• type KubeletPlugin
• type MethodInstance
• type Nodes
  • func NewNodes(f *framework.Framework, minNodes, maxNodes int) *Nodes
  • func NewNodesNow(ctx context.Context, f *framework.Framework, minNodes, maxNodes int) *Nodes
• type Resources

Constants

const (
	NodePrepareResourcesMethod   = "/k8s.io.kubelet.pkg.apis.dra.v1beta1.DRAPlugin/NodePrepareResources"
	NodeUnprepareResourcesMethod = "/k8s.io.kubelet.pkg.apis.dra.v1beta1.DRAPlugin/NodeUnprepareResources"
)

Types

type Driver

type Driver struct {

	// NameSuffix can be set while registering a test to deploy different
	// drivers in the same test namespace.
	NameSuffix string

	// InstanceSuffix can be set while registering a test to deploy two different
	// instances of the same driver. Used to generate unique objects in the API server.
	// The socket path is still the same.
	InstanceSuffix string

	// RollingUpdate can be set to true to enable using different socket names
	// for different pods and thus seamless upgrades. Must be supported by the kubelet!
	RollingUpdate bool

	// Name gets derived automatically from the current test namespace and
	// (if set) the NameSuffix while setting up the driver for a test.
	Name string

	// Nodes contains entries for each node selected for a test when the test runs.
	// In addition, there is one entry for a fictional node.
	Nodes map[string]KubeletPlugin

	NodeV1alpha4 bool
	NodeV1beta1  bool
	// contains filtered or unexported fields
}

func NewDriver

func NewDriver(f *framework.Framework, nodes *Nodes, configureResources func() Resources, devicesPerNode ...map[string]map[resourceapi.QualifiedName]resourceapi.DeviceAttribute) *Driver

NewDriver sets up controller (as client of the cluster) and kubelet plugin (via proxy) before the test runs. It cleans up after the test.

Call this outside of ginkgo.It, then use the instance inside ginkgo.It.

func NewDriverInstance

func NewDriverInstance(f *framework.Framework) *Driver

NewDriverInstance is a variant of NewDriver where the driver is inactive and must be started explicitly with Run. May be used inside ginkgo.It.

func (*Driver) CallCount

func (d *Driver) CallCount(m MethodInstance) int64

func (*Driver) Fail

func (d *Driver) Fail(m MethodInstance, injectError bool)

func (*Driver) IsGone

func (d *Driver) IsGone(ctx context.Context)

IsGone checks that the kubelet is done with the driver. This is done by waiting for the kubelet to remove the driver's ResourceSlices, which takes at least 5 minutes because of the delay in the kubelet. Only use this in slow tests...

func (*Driver) NewGetSlices

func (d *Driver) NewGetSlices() framework.GetFunc[*resourceapi.ResourceSliceList]

NewGetSlices generates a function for gomega.Eventually/Consistently which returns the ResourceSliceList.

func (*Driver) Nodenames

func (d *Driver) Nodenames() (nodenames []string)

func (*Driver) Run

func (d *Driver) Run(nodes *Nodes, configureResources func() Resources, devicesPerNode ...map[string]map[resourceapi.QualifiedName]resourceapi.DeviceAttribute)

func (*Driver) SetUp

func (d *Driver) SetUp(nodes *Nodes, resources Resources, devicesPerNode ...map[string]map[resourceapi.QualifiedName]resourceapi.DeviceAttribute)

func (*Driver) TearDown

func (d *Driver) TearDown(ctx context.Context)

type KubeletPlugin

type KubeletPlugin struct {
	*app.ExamplePlugin
	ClientSet kubernetes.Interface
}

type MethodInstance

type MethodInstance struct {
	Nodename   string
	FullMethod string
}

type Nodes

type Nodes struct {
	NodeNames []string
	// contains filtered or unexported fields
}

func NewNodes

func NewNodes(f *framework.Framework, minNodes, maxNodes int) *Nodes

NewNodes selects nodes to run the test on.

Call this outside of ginkgo.It, then use the instance inside ginkgo.It.

func NewNodesNow

func NewNodesNow(ctx context.Context, f *framework.Framework, minNodes, maxNodes int) *Nodes

NewNodesNow is a variant of NewNodes which can be used inside a ginkgo.It.

type Resources

type Resources struct {
	NodeLocal bool

	// Nodes is a fixed list of node names on which resources are
	// available. Mutually exclusive with NodeLabels.
	Nodes []string

	// Number of devices called "device-000", "device-001", ... on each node or in the cluster.
	MaxAllocations int

	// Tainted causes all devices to be published with a NoExecute taint.
	Tainted bool
}