carina

README

Carina

English | 中文

Background

Storage systems are complex! There are more and more kubernetes native storage systems nowadays and stateful applications are shifting into cloud native world, for example, modern databases and middlewares. However, both modern databases and its storage providers try to solve some common problems in their own way. For example, they both deal with data replications and consistency. This introduces a giant waste of both capacity and performance and needs more mantainness effort. And besides that, stateful applications strive to be more peformant, eliminating every possible latency, which is unavoidable for modern distributed storage systems. Enters carina.

Carina is a standard kubernetes CSI plugin. Users can use standard kubernetes storage resources like storageclass/PVC/PV to request storage media. The key considerations of carina includes:

• Workloads need different storage systems. Carina will focus on cloudnative database scenario usage only.
• Completely kubernetes native and easy to install.
• Using local disks and group them as needed, user can provison different type of disks using different storage class.
• Scaning physical disks and building a RAID as required. If disk fails, just plugin a new one and it's done.
• Node capacity and performance aware, so scheduling pods more smartly.
• Extremly low overhead. Carina sit besides the core data path and provide raw disk performance to applications.
• Auto tiering. Admins can configure carina to combine the large-capacity-but-low-performant disk and small-capacity-but-high-performant disks as one storageclass, so user can benifit both from capacity and performance.
• If nodes fails, carina will automatically detach the local volume from pods thus pods can be rescheduled.
• Middleware runs on baremetals for decades. There are many valueable optimizations and enhancements which are definitely not outdated even in cloudnative era. Let carina be an DBA expert of the storage domain for cloudnative databases!

In short, Carina strives to provide extremely-low-latency and noOps storage system for cloudnative databases and be DBA expert of the storage domain in cloudnative era!

Running Environments

• Kubernetes：(CSI_VERSION=1.5.0)

• Node OS：Linux

• Filesystems：ext4，xfs

• If Kubelet is running in containerized mode, you need to mount the host /dev:/dev directory

• Each node in the cluster has 1..N Bare disks, supporting SSDS and HDDS. (You can run the LSBLK --output NAME,ROTA command to view the disk type. If ROTA=1 is HDD,ROTA =0 is SSD.)

• The capacity of a raw disk must be greater than 10 GB

• If the server does not support the bcache kernel module, see FAQ, Modify yamL deployment

Kubernetes compatiblity

kubernetes	v0.9	v0.9.1	v0.10	v0.11.0	v1.0
>=1.18	support	support	support	support	not released
>=1.25	nonsupport	nonsupport	nonsupport	experimental	not released

Carina architecture

Carina is built for cloudnative stateful applications with raw disk performance and ops-free maintainess. Carina can scan local disks and classify them by disk types， for example, one node can have 10 HDDs and 2 SSDs. Carina then will group them into different disk pools and user can request different disk type by using different storage class. For data HA, carina now leverages STORCLI to build RAID groups.

Carina components

It has three componets: carina-scheduler, carina-controller and carina-node.

• carina-scheduler is an kubernetes scheduler plugin, sorting nodes based on the requested PV size、node's free disk space and node IO perf stats. By default, carina-scheduler supports binpack and spreadout policies.
• carina-controller is the controll plane of carina, which watches PVC resources and maintain the internal logivalVolume object.
• carina-node is an agent which runs on each node. It manage local disks using LVM.

Features

• disk management
• device registration
• volume mode: filesystem
• volume mode: block
• PVC resizing
• scheduing based on capacity
• volume tooplogy
• PVC autotiering
• RAID management
• failover
• io throttling
• metrics
• API

Quickstart

Install by shell

• In this deployment mode, the image TAG is Latest. If you want to deploy a specific version of Carina, you need to change the image address

$ cd deploy/kubernetes
# install， The default installation is kube-system.
$ ./deploy.sh

# uninstall
$ ./deploy.sh uninstall

Install by helm3

• Support installation of specified versions of Carina

helm repo add carina-csi-driver https://carina-io.github.io

helm search repo -l carina-csi-driver

helm install carina-csi-driver carina-csi-driver/carina-csi-driver --namespace kube-system --version v0.11.0

• deployment guide
• user guide

Upgrading

• Uninstall the old version ./deploy.sh uninstall and then install the new version ./deploy.sh (uninstalling carina will not affect volume usage)

Contribution Guide

• development guide
• build local runtime

Blogs

• blogs

Roadmap

• roadmap

Typical storage providers

	NFS/NAS	SAN	Ceph	Carina
typical usage	general storage	high performance block device	extremly scalability	high performance block device for cloudnative applications
filesystem	yes	yes	yes	yes
filesystem type	NFS	driver specific	ext4/xfs	ext4/xfs
block	no	yes	yes	yes
bandwidth	standard	standard	high	high
IOPS	standard	high	standard	high
latency	standard	low	standard	low
CSI support	yes	yes	yes	yes
snapshot	no	driver specific	yes	no
clone	no	driver specific	yes	not yet, comming soon
quota	no	yes	yes	yes
resizing	yes	driver specific	yes	yes
data HA	RAID or NAS appliacne	yes	yes	RAID
ease of maintainess	driver specific	multiple drivers for multiple SAN	high maintainess effort	ops-free
budget	high for NAS	high	high	low, using the extra disks in existing kubernetes cluster
others	data migrates with pods	data migrates with pods	data migrates with pods	binpack or spreadout scheduling policy data doesn't migrate with pods * inplace rebulid if pod fails

FAQ

• FAQ

Similar projects

• openebs
• topolvm
• csi-driver-host-path
• local-path-provisioner

Known Users

Welcome to register the company name in ADOPTERS.md

Community

• For wechat users

License

Carina is under the Apache 2.0 license. See the LICENSE file for details.

Code of Conduct

Please refer to our Carina Community Code of Conduct

Documentation

Index

• Constants

Constants

const (
	// Version project
	Version = "beta"
	// CSIPluginName PluginName is the name of the CSI plugin.
	CSIPluginName = "carina.storage.io"
	// DefaultCSISocket is the default path of the CSI socket file.
	DefaultCSISocket = "/tmp/csi/csi-provisioner.sock"
	// DefaultReservedSpace Default disk space hold
	DefaultReservedSpace = 10 << 30
	DefaultEdgeSpace     = 1 << 30

	// LogicVolumeFinalizer LogicalVolumeFinalizer is the name of LogicalVolume finalizer
	LogicVolumeFinalizer = "carina.storage.io/logicvolume"
	// ResizeRequestedAtKey is the key of LogicalVolume that represents the timestamp of the resize request.
	ResizeRequestedAtKey = "carina.storage.io/resize-requested-at"

	//  ExclusivityDisk  true or false  is the key indicates that only the disk is used by one pod
	ExclusivityDisk = "carina.storage.io/exclusively-raw-disk"

	VolumeManagerType = "carina.io/volume-manage-type"

	// DeviceDiskKey storage class
	// DeviceDiskKey is the key used in CSI volume create requests to specify a DeviceDiskKey support carina-vg-ssd carina-vg-hdd
	DeviceDiskKey = "carina.storage.io/disk-group-name"

	VolumeBackendDiskType = "carina.storage.io/backend-disk-group-name"
	VolumeCacheDiskType   = "carina.storage.io/cache-disk-group-name"
	// VolumeCacheDiskRatio value: 1-100 Cache Capacity Ratio
	VolumeCacheDiskRatio = "carina.storage.io/cache-disk-ratio"
	// VolumeCachePolicy value: writethrough|writeback|writearound
	VolumeCachePolicy = "carina.storage.io/cache-policy"

	// MinRequestSizeGb pvc
	// default size in GiB for volumes (PVC or inline ephemeral volumes) w/o capacity requests.
	MinRequestSizeGb = 1
	// AnnSelectedNode This annotation is added to a PVC that has been triggered by scheduler to
	// be dynamically provisioned. Its value is the name of the selected node.
	AnnSelectedNode = "volume.kubernetes.io/selected-node"

	// VolumeDevicePath pv csi VolumeAttributes
	VolumeDevicePath  = "carina.storage.io/path"
	VolumeDeviceNode  = "carina.storage.io/node"
	VolumeDeviceMajor = "carina.storage.io/major"
	VolumeDeviceMinor = "carina.storage.io/minor"

	VolumeCacheDevicePath  = "carina.storage.io/cache/path"
	VolumeCacheDeviceMajor = "carina.storage.io/cache/major"
	VolumeCacheDeviceMinor = "carina.storage.io/cache/minor"
	VolumeCacheId          = "carina.storage.io/cache-volume-id"
	VolumeCacheBlock       = "carina.storage.io/cache/block"
	VolumeCacheBucket      = "carina.storage.io/cache/bucket"

	// TopologyNodeKey topology
	// TopologyZoneKey is the key of topology that represents zone name.
	TopologyNodeKey = "topology.carina.storage.io/node"

	// DeviceCapacityKeyPrefix device plugin
	DeviceCapacityKeyPrefix = "carina.storage.io/"
	// DeviceVGSSD support disk type
	DeviceVGSSD = "carina-vg-ssd"
	DeviceVGHDD = "carina-vg-hdd"

	// CarinaSchedule custom schedule
	CarinaSchedule = "carina-scheduler"

	// DeviceVolumeType type
	LvmVolumeType = "lvm"
	RawVolumeType = "raw"

	AllowPodMigrationIfNodeNotready = "carina.storage.io/allow-pod-migration-if-node-notready"

	CarinaPrefix = "carina.io"

	ConfigSourceAnnotationKey = "kubernetes.io/config.source"
	// Updates from Kubernetes API Server
	ApiserverSource = "api"

	ThinPrefix   = "thin-"
	VolumePrefix = "volume-"

	ResourceExhausted = "don't have enough space"
)