nri

README

Node Resource Interface, Revisited

This project is currently in DRAFT status

Goal

NRI allows plugging domain- or vendor-specific custom logic into OCI- compatible runtimes. This logic can make controlled changes to containers or perform extra actions outside the scope of OCI at certain points in a containers lifecycle. This can be used, for instance, for improved allocation and management of devices and other container resources.

NRI defines the interfaces and implements the common infrastructure for enabling such pluggable runtime extensions, NRI plugins. This also keeps the plugins themselves runtime-agnostic.

The goal is to enable NRI support in the most commonly used OCI runtimes, containerd and CRI-O.

Background

The revisited API is a major rewrite of NRI. It changes the scope of NRI and how it gets integrated into runtimes. It reworks how plugins are implemented, how they communicate with the runtime, and what kind of changes they can make to containers.

NRI v0.1.0 used an OCI hook-like one-shot plugin invocation mechanism where a separate instance of a plugin was spawned for every NRI event. This instance then used its standard input and output to receive a request and provide a response, both as JSON data.

Plugins in NRI are daemon-like entities. A single instance of a plugin is now responsible for handling the full stream of NRI events and requests. A unix-domain socket is used as the transport for communication. Instead of JSON requests and responses NRI is defined as a formal, protobuf-based 'NRI plugin protocol' which is compiled into ttRPC bindings. This should result in improved communication efficiency with lower per-message overhead, and enable straightforward implementation of stateful NRI plugins.

Components

The NRI implementation consists of a number of components. The core of these are essential for implementing working end-to-end NRI support in runtimes. These core components are the actual NRI protocol, and the NRI runtime adaptation.

Together these establish the model of how a runtime interacts with NRI and how plugins interact with containers in the runtime through NRI. They also define under which conditions plugins can make changes to containers and the extent of these changes.

The rest of the components are the NRI plugin stub library and some sample NRI plugins. Some plugins implement useful functionality in real world scenarios. A few others are useful for debugging. All of the sample plugins serve as practical examples of how the stub library can be used to implement NRI plugins.

Protocol, Plugin API

The core of NRI is defined by a protobuf protocol definition of the low-level plugin API. The API defines two services, Runtime and Plugin.

The Runtime service is the public interface runtimes expose for NRI plugins. All requests on this interface are initiated by the plugin. The interface provides functions for

• initiating plugin registration
• requesting unsolicited updates to containers

The Plugin service is the public interface NRI uses to interact with plugins. All requests on this interface are initiated by NRI/the runtime. The interface provides functions for

• configuring the plugin
• getting initial list of already existing pods and containers
• hooking the plugin into pod/container lifecycle events
• shutting down the plugin

Plugin Registration

Before a plugin can start receiving and processing container events, it needs to register itself with NRI. During registration the plugin and NRI perform a handshake sequence which consists of the following steps:

1. the plugin identifies itself to the runtime
2. NRI provides plugin-specific configuration data to the plugin
3. the plugin subscribes to pod and container lifecycle events of interest
4. NRI sends list of existing pods and containers to plugin
5. the plugin requests any updates deemed necessary to existing containers

The plugin identifies itself to NRI by a plugin name and a plugin index. The plugin index is used by NRI to determine in which order the plugin is hooked into pod and container lifecycle event processing with respect to any other plugins.

The plugin name is used to pick plugin-specific data to send to the plugin as configuration. This data is only present if the plugin has been launched by NRI. If the plugin has been externally started it is expected to acquire its configuration also by external means. The plugin subscribes to pod and container lifecycle events of interest in its response to configuration.

As the last step in the registration and handshaking process, NRI sends the full set of pods and containers known to the runtime. The plugin can request updates it considers necessary to any of the known containers in response.

Once the handshake sequence is over and the plugin has registered with NRI, it will start receiving pod and container lifecycle events according to its subscription.

Pod Data and Available Lifecycle Events

NRI Pod Lifecycle Events

NRI plugins can subscribe to the following pod lifecycle events:

• creation
• stopping
• removal

The following pieces of pod metadata are available to plugins in NRI:

• ID
• name
• UID
• namespace
• labels
• annotations
• cgroup parent directory
• runtime handler name

Container Data and Available Lifecycle Events

NRI Container Lifecycle Events

NRI plugins can subscribe to the following container lifecycle events:

• creation (*)
• post-creation
• starting
• post-start
• updating (*)
• post-update
• stopping (*)
• removal

*) Plugins can request adjustment or updates to containers in response to these events.

The following pieces of container metadata are available to plugins in NRI:

• ID
• pod ID
• name
• state
• labels
• annotations
• command line arguments
• environment variables
• mounts
• OCI hooks
• rlimits
• linux
  • namespace IDs
  • devices
  • resources
    • memory
      • limit
      • reservation
      • swap limit
      • kernel limit
      • kernel TCP limit
      • swappiness
      • OOM disabled flag
      • hierarchical accounting flag
      • hugepage limits
    • CPU
      • shares
      • quota
      • period
      • realtime runtime
      • realtime period
      • cpuset CPUs
      • cpuset memory
    • Block I/O class
    • RDT class

Apart from data identifying the container, these pieces of information represent the corresponding data in the container's OCI Spec.

Container Adjustment

During container creation plugins can request changes to the following container parameters:

• annotations
• mounts
• environment variables
• OCI hooks
• rlimits
• linux
  • devices
  • resources
    • memory
      • limit
      • reservation
      • swap limit
      • kernel limit
      • kernel TCP limit
      • swappiness
      • OOM disabled flag
      • hierarchical accounting flag
      • hugepage limits
    • CPU
      • shares
      • quota
      • period
      • realtime runtime
      • realtime period
      • cpuset CPUs
      • cpuset memory
    • Block I/O class
    • RDT class

Container Updates

Once a container has been created plugins can request updates to them. These updates can be requested in response to another containers creation request, in response to any containers update request, in response to any containers stop request, or they can be requested as part of a separate unsolicited container update request. The following container parameters can be updated this way:

• resources
  • memory
    • limit
    • reservation
    • swap limit
    • kernel limit
    • kernel TCP limit
    • swappiness
    • OOM disabled flag
    • hierarchical accounting flag
    • hugepage limits
  • CPU
    • shares
    • quota
    • period
    • realtime runtime
    • realtime period
    • cpuset CPUs
    • cpuset memory
  • Block I/O class
  • RDT class

Runtime Adaptation

The NRI runtime adaptation package is the interface runtimes use to integrate to NRI and interact with NRI plugins. It implements basic plugin discovery, startup and configuration. It also provides the functions necessary to hook NRI plugins into lifecycle events of pods and containers from the runtime.

The package hides the fact that multiple NRI plugins might be processing any single pod or container lifecycle event. It takes care of invoking plugins in the correct order and combining responses by multiple plugins into a single one. While combining responses, the package detects any unintentional conflicting changes made by multiple plugins to a single container and flags such an event as an error to the runtime.

Wrapped OCI Spec Generator

The OCI Spec generator package wraps the corresponding package and adds functions for applying NRI container adjustments and updates to OCI Specs. This package can be used by runtime NRI integration code to apply NRI responses to containers.

Plugin Stub Library

The plugin stub hides many of the low-level details of implementing an NRI plugin. It takes care of connection establishment, plugin registration, configuration, and event subscription. All sample plugins are implemented using the stub. Any of these can be used as a tutorial on how the stub library should be used.

Sample Plugins

The following sample plugins exist for NRI:

• logger
• differ
• device injector
• OCI hook injector
• ulimit adjuster
• NRI v0.1.0 plugin adapter

Please see the documentation of these plugins for further details about what and how each of these plugins can be used for.

Security Considerations

From a security perspective NRI plugins should be considered part of the container runtime. NRI does not implement granular access control to the functionality it offers. Access to NRI is controlled by restricting access to the systemwide NRI socket. If a process can connect to the NRI socket and send data, it has access to the full scope of functionality available via NRI.

In particular this includes

• injection of OCI hooks, which allow for arbitrary execution of processes with the same privilege level as the container runtime
• arbitrary changes to mounts, including new bind-mounts, changes to the proc, sys, mqueue, shm, and tmpfs mounts
• the addition or removal of arbitrary devices
• arbitrary changes to the limits for memory, CPU, block I/O, and RDT resources available, including the ability to deny service by setting limits very low

The same precautions and principles apply to protecting the NRI socket as to protecting the socket of the runtime itself. Unless it already exists, NRI itself creates the directory to hold its socket with permissions that allow access only for the user ID of the runtime process. By default this limits NRI access to processes running as root (UID 0). Changing the default socket permissions is strongly advised against. Enabling more permissive access control to NRI should never be done without fully understanding the full implications and potential consequences to container security.

Plugins as Kubernetes DaemonSets

When the runtime manages pods and containers in a Kubernetes cluster, it is convenient to deploy and manage NRI plugins using Kubernetes DaemonSets. Among other things, this requires bind-mounting the NRI socket into the filesystem of a privileged container running the plugin. Similar precautions apply and the same care should be taken for protecting the NRI socket and NRI plugins as for the kubelet DeviceManager socket and Kubernetes Device Plugins.

The cluster configuration should make sure that unauthorized users cannot bind-mount host directories and create privileged containers which gain access to these sockets and can act as NRI or Device Plugins. See the related documentation and best practices about Kubernetes security.

API Stability

NRI APIs should not be considered stable yet. We try to avoid unnecessarily breaking APIs, especially the Stub API which plugins use to interact with NRI. However, before NRI reaches a stable 1.0.0 release, this is only best effort and cannot be guaranteed. Meanwhile we do our best to document any API breaking changes for each release in the release notes.

The current target for a stable v1 API through a 1.0.0 release is the end of this year.

Project details

nri is a containerd sub-project, licensed under the Apache 2.0 license. As a containerd sub-project, you will find the:

• Project governance,
• Maintainers,
• and Contributing guidelines

information in our containerd/project repository.

Documentation

Index

• Constants
• type Client
  • func New() (*Client, error)
  • func (c *Client) Invoke(ctx context.Context, task Task, state types.State) ([]*types.Result, error)
  • func (c *Client) InvokeWithSandbox(ctx context.Context, task Task, state types.State, sandbox *Sandbox) ([]*types.Result, error)
• type Sandbox
• type Task

Constants

const (
	// DefaultBinaryPath for nri plugins
	DefaultBinaryPath = "/opt/nri/bin"
	// DefaultConfPath for the global nri configuration
	DefaultConfPath = "/etc/nri/conf.json"
	// Version of NRI
	Version = "0.1"
)

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client for calling nri plugins

func New

func New() (*Client, error)

New nri client

func (*Client) Invoke

func (c *Client) Invoke(ctx context.Context, task Task, state types.State) ([]*types.Result, error)

Invoke the ConfList of nri plugins

func (*Client) InvokeWithSandbox

func (c *Client) InvokeWithSandbox(ctx context.Context, task Task, state types.State, sandbox *Sandbox) ([]*types.Result, error)

InvokeWithSandbox invokes the ConfList of nri plugins

type Sandbox

type Sandbox struct {
	// ID of the sandbox
	ID string
	// Labels of the sandbox
	Labels map[string]string
}

Sandbox information

type Task

type Task interface {

	// Spec returns the current OCI specification for the task
	Spec(context.Context) (*oci.Spec, error)
	// contains filtered or unexported methods
}

Task is a subset of containerd's Task interface.