v1.12.1 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2020 License: Apache-2.0 Imports: 32 Imported by: 1



Package app contains OpenEBS Dynamic Local PV provisioner

Provisioner is created using the external storage provisioner library:

Local PVs are an extension to hostpath volumes, but are more secure.

Local PVs are great in cases like:

  • The Stateful Workload can take care of replicating the data across nodes to handle cases like a complete node (and/or its storage) failure.
  • For long running Stateful Workloads, the Backup/Recovery is provided by Operators/tools that can make use the Workload mounts and do not require the capabilities to be available in the underlying storage. Or if the hostpaths are created on external storage like EBS/GPD, administrator have tools that can periodically take snapshots/backups.

OpenEBS Local PVs extends the capabilities provided by the Kubernetes Local PV by making use of the OpenEBS Node Storage Device Manager (NDM), the significant differences include:

  • Users need not pre-format and mount the devices in the node.
  • Supports Dynamic Local PVs - where the devices can be used by CAS solutions and also by applications. CAS solutions typically directly access a device. OpenEBS Local PV ease the management of storage devices to be used between CAS solutions (direct access) and applications (via PV), by making use of BlockDeviceClaims supported by OpenEBS NDM.
  • Supports using hostpath as well for provisioning a Local PV. In fact in some cases, the Kubernetes nodes may have limited number of storage devices attached to the node and hostpath based Local PVs offer efficient management of the storage available on the node.

Inspiration: ------------ OpenEBS Local PV has been inspired by the prior work done by the following the Kubernetes projects: - - -

How it works: ------------- Step 1: Multiple Storage Classes can be created by the Kubernetes Administrator, to specify the required type of OpenEBS Local PV to be used by an application. A simple StorageClass looks like:

--- apiVersion: kind: StorageClass metadata:

name: hostpath
  #Define a new OpenEBS CAS Type called `local`
  #which indicates that Data is stored
  #directly onto hostpath. The hostpath can be:
  #- device (as block or mounted path)
  #- hostpath (sub directory on OS or mounted path) local |
    #- name: StorageType
    #  value: "device"
    # (Default)
    - name: StorageType
      value: "hostpath"
    # If the StorageType is hostpath, then BasePath
    # specifies the location where the volume sub-directory
    # should be created.
    # (Default)
    - name: BasePath
      value: "/var/openebs/local"

provisioner: volumeBindingMode: WaitForFirstConsumer reclaimPolicy: Delete ---

Step 2: The application developers will request for storage via PVC as follows: --- apiVersion: v1 kind: PersistentVolumeClaim metadata:

name: pvc-hp


  - ReadWriteOnce
storageClassName: hostpath
    storage: 2Gi


Step 3: A Local PV (type=hostpath) provisioned via the OpenEBS Dynamic Local PV Provisioner looks like this: --- apiVersion: v1 kind: PersistentVolume metadata:

creationTimestamp: 2019-05-02T15:44:35Z
name: pvc-2fe08284-6cf1-11e9-be8b-42010a800155
resourceVersion: "2062"
selfLink: /api/v1/persistentvolumes/pvc-2fe08284-6cf1-11e9-be8b-42010a800155
uid: 2fedaff8-6cf1-11e9-be8b-42010a800155


- ReadWriteOnce
  storage: 2Gi
  apiVersion: v1
  kind: PersistentVolumeClaim
  name: pvc-hp
  namespace: default
  resourceVersion: "2060"
  uid: 2fe08284-6cf1-11e9-be8b-42010a800155
  path: /var/openebs/local/pvc-2fe08284-6cf1-11e9-be8b-42010a800155
  fsType: ""
    - matchExpressions:
      - key:
        operator: In
        - gke-kmova-helm-default-pool-6c1271a5-n8b0
persistentVolumeReclaimPolicy: Delete
storageClassName: hostpath


phase: Bound


Note that the location of the hostpaths on the node are abstracted from the application developers and are under the Administrators control.

Implementation Details: ----------------------- (a) The configuration of whether to select a complete storage device

of a hostpath is determined by the StorageClass annotations, inline
with other configuration options provided by OpenEBS.

(b) When using StorageType as device, the Local Provisioner will

interact with the OpenEBS Node Storage Device Manager (NDM) to
identify the device to be used. Each PVC of type StorageType=device
will create a BDC and wait for the NDM to provide an appropriate BD.
From the BD, the provisioner will extract the path details and create
a Local PV. When using unformatted block devices, the administrator
can specific the type of FS to be put on block devices using the
FSType CAS Policy in StorageClass.

(c) The StorageClass can work either with `waitForConsumer`, in which

case, the PV is created on the node where the Pod is scheduled or
vice versa. (Note: In the initial version, only `waitForConsumer` is

(d) When using the hostpath, the administrator can select the location using:

  • BasePath: By default, the hostpath volumes will be created under `/var/openebs/local`. This default path can be changed by passing the "OPENEBS_IO_BASE_PATH" ENV variable to the Hostpath Provisioner Pod. It is also possible to specify a different location using the CAS Policy `BasePath` in the StorageClass.

    The hostpath used in the above configuration can be:

  • OS Disk - possibly a folder dedicated to saving data on each node.

  • Additional Disks - mounted as ext4 or any other filesystem

  • External Storage - mounted as ext4 or any other filesystem

(e) The backup and restore via Velero Plugin has been verified to work for

OpenEBS Local PV. Supported from OpenEBS 1.0 and higher.

Future Improvements and Limitations: ------------------------------------

  • Ability to enforce capacity limits. The application can exceed it usage of capacity beyond what it requested.
  • Ability to enforce provisioning limits based on the capacity available on a given node or the number of PVs already provisioned.
  • Ability to use hostpaths and devices that can potentially support snapshots. Example: a hostpath backed by github, or by LVM or ZFS where capacity also can be enforced.
  • Extend the capabilities of the Local PV provisioner to handle cases where underlying devices are moved to new node and needs changes to the node affinity.
  • Move towards using a CSI based Hostpath provisioner.



View Source
const (
	//KeyPVStorageType defines if the PV should be backed
	// a hostpath ( sub directory or a storage device)
	KeyPVStorageType = "StorageType"

	//KeyPVBasePath defines base directory for hostpath volumes
	// can be configured via the StorageClass annotations.
	KeyPVBasePath = "BasePath"

	//KeyPVFSType defines filesystem type to be used with devices
	// and can be configured via the StorageClass annotations.
	KeyPVFSType = "FSType"

	//KeyBDTag defines the value for the Block Device Tag
	//label selector configured via the StorageClass annotations.
	//User can group block devices across nodes by setting the
	//label on block devices as:
	//The <tag-value> used above can be passsed to the
	//Local PV device provisioner via the StorageClass
	//CAS annotations, to specify that Local PV (device)
	//should only make use of those block devices that
	//tagged with the given <tag-value>.
	//Example: Local PV device StorageClass for picking devices
	//labeled as:
	//will be as follows
	// kind: StorageClass
	// metadata:
	//   name: openebs-device-tag-x
	//   annotations:
	// local
	// |
	//       - name: StorageType
	//         value: "device"
	//       - name: BlockDeviceTag
	//         value: "tag-x"
	// provisioner:
	// volumeBindingMode: WaitForFirstConsumer
	// reclaimPolicy: Delete
	KeyBDTag = "BlockDeviceTag"
View Source
const (
	// ProvisionerHelperImage is the environment variable that provides the
	// container image to be used to launch the help pods managing the
	// host path
	ProvisionerHelperImage menv.ENVKey = "OPENEBS_IO_HELPER_IMAGE"

	// ProvisionerBasePath is the environment variable that provides the
	// default base path on the node where host-path PVs will be provisioned.
	ProvisionerBasePath menv.ENVKey = "OPENEBS_IO_BASE_PATH"
View Source
const (

	// LocalPVFinalizer represents finalizer string used by LocalPV
	LocalPVFinalizer = ""


View Source
var (
	//CmdTimeoutCounts specifies the duration to wait for cleanup pod
	//to be launched.
	CmdTimeoutCounts = 120
View Source
var (

	// LeaderElectionKey represents ENV for disable/enable leaderElection for
	// localpv provisioner
View Source
var (
	//WaitForBDTimeoutCounts specifies the duration to wait for BDC to be associated with a BD
	//The duration is the value specified here multiplied by 5
	WaitForBDTimeoutCounts = 12



func GetLocalPVType

func GetLocalPVType(pv *v1.PersistentVolume) string

GetLocalPVType extracts the Local PV Type from PV

func GetNodeHostname

func GetNodeHostname(n *v1.Node) string

GetNodeHostname extracts the Hostname from the labels on the Node If hostname label `` is not present an empty string is returned.

func GetStorageClassName

func GetStorageClassName(pvc *v1.PersistentVolumeClaim) *string

GetStorageClassName extracts the StorageClass name from PVC

func GetTaints

func GetTaints(n *v1.Node) []v1.Taint

GetTaints extracts the Taints from the Spec on the node If Taints are empty, it just returns empty structure of corev1.Taints

func RegisterShutdownChannel

func RegisterShutdownChannel(done chan struct{})

RegisterShutdownChannel closes the channel when signaled for termination

func Start

func Start(cmd *cobra.Command) error

Start will initialize and run the dynamic provisioner daemon

func StartProvisioner

func StartProvisioner() (*cobra.Command, error)

StartProvisioner will start a new dynamic Host Path PV provisioner


type GetVolumeConfigFn

type GetVolumeConfigFn func(pvName string, pvc *v1.PersistentVolumeClaim) (*VolumeConfig, error)

GetVolumeConfigFn allows to plugin a custom function

and makes it easy to unit test provisioner

type HelperBlockDeviceOptions

type HelperBlockDeviceOptions struct {
	// contains filtered or unexported fields

HelperBlockDeviceOptions contains the options that will launch a BDC on a specific node (nodeHostname)

type HelperPodOptions

type HelperPodOptions struct {
	// contains filtered or unexported fields

HelperPodOptions contains the options that will launch a Pod on a specific node (nodeHostname) to execute a command (cmdsForPath) on a given volume path (path)

type Provisioner

type Provisioner struct {
	// contains filtered or unexported fields

Provisioner struct has the configuration and utilities required across the different work-flows.

func NewProvisioner

func NewProvisioner(stopCh chan struct{}, kubeClient *clientset.Clientset) (*Provisioner, error)

NewProvisioner will create a new Provisioner object and initialize

it with global information used across PV create and delete operations.

func (*Provisioner) Delete

func (p *Provisioner) Delete(pv *v1.PersistentVolume) (err error)

Delete is invoked by the PVC controller to perform clean-up

activities before deleteing the PV object. If reclaim policy is
set to not-retain, then this function will create a helper pod
to delete the host path from the node.

func (*Provisioner) DeleteBlockDevice

func (p *Provisioner) DeleteBlockDevice(pv *v1.PersistentVolume) (err error)

DeleteBlockDevice is invoked by the PVC controller to perform clean-up

activities before deleteing the PV object. If reclaim policy is
set to not-retain, then this function will delete the associated BDC

func (*Provisioner) DeleteHostPath

func (p *Provisioner) DeleteHostPath(pv *v1.PersistentVolume) (err error)

DeleteHostPath is invoked by the PVC controller to perform clean-up

activities before deleteing the PV object. If reclaim policy is
set to not-retain, then this function will create a helper pod
to delete the host path from the node.

func (*Provisioner) GetNodeObjectFromHostName

func (p *Provisioner) GetNodeObjectFromHostName(hostName string) (*v1.Node, error)

GetNodeObjectFromHostName returns the Node Object with matching NodeHostName.

func (*Provisioner) GetVolumeConfig

func (p *Provisioner) GetVolumeConfig(pvName string, pvc *v1.PersistentVolumeClaim) (*VolumeConfig, error)

GetVolumeConfig creates a new VolumeConfig struct by parsing and merging the configuration provided in the PVC annotation - with the default configuration of the provisioner.

func (*Provisioner) Provision

Provision is invoked by the PVC controller which expect the PV

to be provisioned and a valid PV spec returned.

func (*Provisioner) ProvisionBlockDevice

func (p *Provisioner) ProvisionBlockDevice(opts pvController.ProvisionOptions, volumeConfig *VolumeConfig) (*v1.PersistentVolume, error)

ProvisionBlockDevice is invoked by the Provisioner to create a Local PV

with a Block Device

func (*Provisioner) ProvisionHostPath

func (p *Provisioner) ProvisionHostPath(opts pvController.ProvisionOptions, volumeConfig *VolumeConfig) (*v1.PersistentVolume, error)

ProvisionHostPath is invoked by the Provisioner which expect HostPath PV

to be provisioned and a valid PV spec returned.

func (*Provisioner) SupportsBlock

func (p *Provisioner) SupportsBlock() bool

SupportsBlock will be used by controller to determine if block mode is

supported by the host path provisioner.

type VolumeConfig

type VolumeConfig struct {
	// contains filtered or unexported fields

VolumeConfig struct contains the merged configuration of the PVC and the associated SC. The configuration is derived from the annotation ``. The configuration will be in the following json format:

	enabled: true
	value: "string value"
	enabled: true
	value: "string value"

func (*VolumeConfig) GetBDTagValue

func (c *VolumeConfig) GetBDTagValue() string

GetBDTagValue returns the block device tag value configured in StorageClass.

Default is "", no device tag will be set and any available block device (without labelled with tag) can be used for creating Local PV(device).

func (*VolumeConfig) GetFSType

func (c *VolumeConfig) GetFSType() string

GetFSType returns the FSType value configured in StorageClass. Default is "", auto-determined by Local PV

func (*VolumeConfig) GetPath

func (c *VolumeConfig) GetPath() (string, error)

GetPath returns a valid PV path based on the configuration or an error. The Path is constructed using the following rules: If AbsolutePath is specified return it. (Future) If PVPath is specified, suffix it with BasePath and return it. (Future) If neither of above are specified, suffix the PVName to BasePath

and return it

Also before returning the path, validate that path is safe

and matches the filters specified in StorageClass.

func (*VolumeConfig) GetStorageType

func (c *VolumeConfig) GetStorageType() string

GetStorageType returns the StorageType value configured in StorageClass. Default is hostpath

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL