cke

package module
v1.25.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 19, 2023 License: Apache-2.0 Imports: 57 Imported by: 0

README

GitHub release CI PkgGoDev Go Report Card CII Best Practices

Cybozu Kubernetes Engine

Kubernetes certification logo

CKE (Cybozu Kubernetes Engine) is a distributed service that automates Kubernetes cluster management.

Project Status: GA

Requirements

CKE requirements
Node OS Requirements
  • Docker: etcd data is stored in Docker volumes.
  • A user who belongs to docker group
  • SSH access for the user

Features

  • Bootstrapping and life-cycle management.

    CKE can bootstrap a Kubernetes and etcd cluster from scratch. CKE can also add or remove nodes to/from the Kubernetes and etcd cluster.

  • In-place and fast upgrade of Kubernetes

    A version of CKE corresponds strictly to a single version of Kubernetes. Therefore, upgrading CKE will upgrade the managed Kubernetes.

    Unlike kubeadm or similar tools, CKE can automatically upgrade its managed Kubernetes without draining nodes. The time taken for the upgrade is not proportional to the number of nodes, so it is very fast.

  • Graceful rebooting of nodes

    CKE can reboot specified nodes gracefully using the Kubernetes eviction API.

  • Managed etcd cluster

    CKE manages an etcd cluster for Kubernetes. Other applications may also store their data in the same etcd cluster.

    Details are described in docs/etcd.md.

  • CRI runtimes

    In addition to Docker, CRI runtimes such as containerd or cri-o can be used to run Kubernetes Pods.

  • Certificate for admission webhooks

    Admission webhooks are Kubernetes extension to validate or mutate API resources. Installing them requires some sort of self-signed X509 certificates.

    CKE can become a certificate authority (CA) and issue certificates for these webhooks.

  • Kubernetes features:

  • User-defined resources:

    CKE automatically creates or updates Kubernetes API resources such as Deployments, Namespaces, or CronJobs that are defined by users. This feature helps users to automate Kubernetes cluster maintenance.

  • Sabakan integration

    CKE can be integrated with sabakan, a service that automates physical server management, to generate cluster configuration automatically.

    Sabakan is not a requirement; cluster configuration can be supplied externally by a YAML file.

  • High availability

    CKE stores its configurations in etcd to share them among multiple instances. Etcd is also used to elect a leader instance that exclusively controls the Kubernetes cluster.

  • Operation logs

    To track problems and life-cycle events, CKE keeps operation logs in etcd.

Programs

This repository contains these programs:

  • cke: the service.
  • ckecli: CLI tool for cke.
  • cke-localproxy: an optional service to run kube-proxy on the same host as CKE.

To see their usage, run them with -h option.

Getting started

A demonstration of CKE running on docker is available at example directory.

Documentation

docs directory contains tutorials and specifications.

Usage

Run CKE with docker
$ docker run -d --read-only \
    --network host --name cke \
    quay.io/cybozu/cke:1.18 [options...]
Install ckecli and cke-localproxy to a host directory
$ docker run --rm -u root:root \
    --entrypoint /usr/local/cke/install-tools \
    --mount type=bind,src=DIR,target=/host \
    quay.io/cybozu/cke:1.18

Docker images

Docker images are available on Quay.io

Feedback

Please report bugs / issues to GitHub issues.

Feel free to send your pull requests!

License

CKE is licensed under the Apache License, Version 2.0.

Documentation

Index

Constants

View Source 
const (
	PropagationShared   = BindPropagation("shared")
	PropagationSlave    = BindPropagation("slave")
	PropagationPrivate  = BindPropagation("private")
	PropagationRShared  = BindPropagation("rshared")
	PropagationRSlave   = BindPropagation("rslave")
	PropagationRPrivate = BindPropagation("rprivate")
)

Bind propagation definitions

View Source 
const (
	LabelShared  = SELinuxLabel("z")
	LabelPrivate = SELinuxLabel("Z")
)

SELinux Label definitions

View Source 
const (
	ProxyModeUserspace proxyv1alpha1.ProxyMode = "userspace"
	ProxyModeIptables  proxyv1alpha1.ProxyMode = "iptables"
	ProxyModeIPVS      proxyv1alpha1.ProxyMode = "ipvs"
)
View Source 
const (
	EtcdImage            = Image("quay.io/cybozu/etcd:3.5.7.2")
	KubernetesImage      = Image("quay.io/cybozu/kubernetes:1.25.10.1")
	ToolsImage           = Image("quay.io/cybozu/cke-tools:1.25.0")
	PauseImage           = Image("quay.io/cybozu/pause:3.8.0.2")
	CoreDNSImage         = Image("quay.io/cybozu/coredns:1.10.0.2")
	UnboundImage         = Image("quay.io/cybozu/unbound:1.17.1.3")
	UnboundExporterImage = Image("quay.io/cybozu/unbound_exporter:0.4.1.5")
)

Container image definitions

View Source 
const (
	PhaseUpgradeAborted  = OperationPhase("upgrade-aborted")
	PhaseUpgrade         = OperationPhase("upgrade")
	PhaseRivers          = OperationPhase("rivers")
	PhaseEtcdBootAborted = OperationPhase("etcd-boot-aborted")
	PhaseEtcdBoot        = OperationPhase("etcd-boot")
	PhaseEtcdStart       = OperationPhase("etcd-start")
	PhaseEtcdWait        = OperationPhase("etcd-wait")
	PhaseK8sStart        = OperationPhase("k8s-start")
	PhaseEtcdMaintain    = OperationPhase("etcd-maintain")
	PhaseK8sMaintain     = OperationPhase("k8s-maintain")
	PhaseStopCP          = OperationPhase("stop-control-plane")
	PhaseUncordonNodes   = OperationPhase("uncordon-nodes")
	PhaseRebootNodes     = OperationPhase("reboot-nodes")
	PhaseCompleted       = OperationPhase("completed")
)

Processing statuses of CKE server.

View Source 
const (
	CAServer                = "server"
	CAEtcdPeer              = "etcd-peer"
	CAEtcdClient            = "etcd-client"
	CAKubernetes            = "kubernetes"
	CAKubernetesAggregation = "kubernetes-aggregation"
	CAWebhook               = "kubernetes-webhook"
)

CA keys for etcd storage.

View Source 
const (
	RoleSystem                = "system"
	RoleAdmin                 = "admin"
	RoleKubeScheduler         = "kube-scheduler"
	RoleKubeControllerManager = "kube-controller-manager"
	RoleKubelet               = "kubelet"
	RoleKubeProxy             = "kube-proxy"
	RoleServiceAccount        = "service-account"
)

Role name in Vault

View Source 
const (
	RebootStatusQueued    = RebootStatus("queued")
	RebootStatusDraining  = RebootStatus("draining")
	RebootStatusRebooting = RebootStatus("rebooting")
	RebootStatusCancelled = RebootStatus("cancelled")
)

Reboot statuses

View Source 
const (
	StatusNew       = RecordStatus("new")
	StatusRunning   = RecordStatus("running")
	StatusCancelled = RecordStatus("cancelled")
	StatusCompleted = RecordStatus("completed")
)

Record statuses

View Source 
const (
	AnnotationResourceImage     = "cke.cybozu.com/image"
	AnnotationResourceRevision  = "cke.cybozu.com/revision"
	AnnotationResourceInjectCA  = "cke.cybozu.com/inject-cacert"
	AnnotationResourceIssueCert = "cke.cybozu.com/issue-cert"
)

Annotations for CKE-managed resources.

View Source 
const (
	KindDeployment                     = "Deployment"
	KindMutatingWebhookConfiguration   = "MutatingWebhookConfiguration"
	KindSecret                         = "Secret"
	KindValidatingWebhookConfiguration = "ValidatingWebhookConfiguration"
)

kinds

View Source 
const (
	KeyCA                    = "ca/"
	KeyConfigVersion         = "config-version"
	KeyCluster               = "cluster"
	KeyClusterRevision       = "cluster-revision"
	KeyConstraints           = "constraints"
	KeyLeader                = "leader/"
	KeyRebootsDisabled       = "reboots/disabled"
	KeyRebootsPrefix         = "reboots/data/"
	KeyRebootsWriteIndex     = "reboots/write-index"
	KeyRecords               = "records/"
	KeyRecordID              = "records"
	KeyResourcePrefix        = "resource/"
	KeySabakanDisabled       = "sabakan/disabled"
	KeySabakanQueryVariables = "sabakan/query-variables"
	KeySabakanTemplate       = "sabakan/template"
	KeySabakanURL            = "sabakan/url"
	KeyServiceAccountCert    = "service-account/certificate"
	KeyServiceAccountKey     = "service-account/key"
	KeyStatus                = "status"
	KeyVault                 = "vault"
)

etcd keys and prefixes

View Source 
const AdminGroup = "system:masters"

AdminGroup is the group name of cluster admin users

View Source 
const (
	// CKELabelName is the name of a Docker label used by CKE.
	CKELabelName = "com.cybozu.cke"
)
View Source 
const CKESecret = "cke/secrets"

CKESecret is the path of key-value secret engine for CKE.

View Source 
const CNAPIServer = "front-proxy-client"

CNAPIServer is the common name of API server for aggregation

View Source 
const ConfigVersion = "2"

ConfigVersion represents the current configuration scheme of how CKE constructs its Kubernetes cluster.

View Source 
const DefaultMaxConcurrentReboots = 1
View Source 
const DefaultRebootEvictionTimeoutSeconds = 600
View Source 
const (

	// DefaultRunTimeout is the timeout value for Agent.Run().
	DefaultRunTimeout = 10 * time.Minute
)
View Source 
const K8sSecret = CKESecret + "/k8s"

K8sSecret is the path of encryption keys used for Kubernetes Secrets.

View Source 
const SSHSecret = CKESecret + "/ssh"

SSHSecret is the path of SSH private keys in Vault.

View Source 
const Version = "1.25.3"

Version represents current cke version

Variables

View Source 
var (
	// ErrNotFound may be returned by Storage methods when a key is not found.
	ErrNotFound = errors.New("not found")
	// ErrNoLeader is returned when the session lost leadership.
	ErrNoLeader = errors.New("lost leadership")
)
View Source 
var AllOperationPhases = []OperationPhase{
	PhaseUpgradeAborted,
	PhaseUpgrade,
	PhaseRivers,
	PhaseEtcdBootAborted,
	PhaseEtcdBoot,
	PhaseEtcdStart,
	PhaseEtcdWait,
	PhaseK8sStart,
	PhaseEtcdMaintain,
	PhaseK8sMaintain,
	PhaseStopCP,
	PhaseUncordonNodes,
	PhaseRebootNodes,
	PhaseCompleted,
}

AllOperationPhases contains all kinds of OperationPhases.

View Source 
var CAKeys = []string{
	CAServer,
	CAEtcdPeer,
	CAEtcdClient,
	CAKubernetes,
	CAKubernetesAggregation,
	CAWebhook,
}

CAKeys is list of CA keys

Functions

func AddUserRole 

func AddUserRole(ctx context.Context, cli *clientv3.Client, name, prefix string) error

AddUserRole create etcd user and role.

func AllImages 

func AllImages() []string

AllImages return container images list used by CKE

func ApplyResource added in v1.13.9 

func ApplyResource(ctx context.Context, dynclient dynamic.Interface, mapper meta.RESTMapper, inf Infrastructure, data []byte, rev int64, forceConflicts bool) error

ApplyResource creates or updates given resource using server-side-apply.

func BuildNodeRebootStatus added in v1.24.1 

func BuildNodeRebootStatus(nodes []*Node, entries []*RebootQueueEntry) map[string]map[string]bool

func ConnectVault 

func ConnectVault(ctx context.Context, data []byte) error

ConnectVault unmarshal data to get VaultConfig and call VaultClient with it. It then start renewing login token for long-running process.

func CountRebootQueueEntries added in v1.22.8 

func CountRebootQueueEntries(entries []*RebootQueueEntry) map[string]int

func GetUserRoles 

func GetUserRoles(ctx context.Context, cli *clientv3.Client, user string) ([]string, error)

GetUserRoles get roles of target user.

func IssueEtcdClientCertificate 

func IssueEtcdClientCertificate(inf Infrastructure, username, ttl string) (cert, key string, err error)

IssueEtcdClientCertificate issues TLS client certificate for a user.

func Kubeconfig 

func Kubeconfig(cluster, user, ca, clientCrt, clientKey string) *api.Config

Kubeconfig creates *api.Config that will be rendered as "kubeconfig" file.

func NewEtcdConfig 

func NewEtcdConfig() *etcdutil.Config

NewEtcdConfig creates Config with default prefix.

func ParseResource added in v1.13.9 

func ParseResource(data []byte) (string, error)

ParseResource parses YAML string.

func SortResources added in v1.13.10 

func SortResources(res []ResourceDefinition)

SortResources sort resources as defined order of creation.

func UserKubeconfig added in v1.14.14 

func UserKubeconfig(cluster, userName, ca, clientCrt, clientKey, server string) *api.Config

UserKubeconfig makes kubeconfig for users

func ValidateProxyMode added in v1.20.0 

func ValidateProxyMode(mode proxyv1alpha1.ProxyMode) error

ValidateProxyMode validates ProxyMode

func VaultClient 

func VaultClient(cfg *VaultConfig) (*vault.Client, *vault.Secret, error)

VaultClient creates vault client. The client has logged-in to Vault using RoleID and SecretID in cfg.

func VaultPKIKey added in v1.18.1 

func VaultPKIKey(caKey string) string

VaultPKIKey returns a key string for Vault corresponding to a CA.

Types

type APIServerParams added in v1.13.3 

type APIServerParams struct {
	ServiceParams   `json:",inline"`
	AuditLogEnabled bool   `json:"audit_log_enabled"`
	AuditLogPolicy  string `json:"audit_log_policy"`
	AuditLogPath    string `json:"audit_log_path"`
}

APIServerParams is a set of extra parameters for kube-apiserver.

type Agent 

type Agent interface {
	// Close closes the underlying connection.
	Close() error

	// Run command on the node.
	// It returns non-nil error if the command takes too long (> DefaultRunTimeout).
	Run(command string) (stdout, stderr []byte, err error)

	// RunWithInput run command with input as stdin.
	// It returns non-nil error if the command takes too long (> DefaultRunTimeout).
	RunWithInput(command, input string) error

	// RunWithTimeout run command with given timeout.
	// If timeout is 0, the command will run indefinitely.
	RunWithTimeout(command, input string, timeout time.Duration) (stdout, stderr []byte, err error)
}

Agent is the interface to run commands on a node.

func SSHAgent 

func SSHAgent(node *Node, privkey string) (Agent, error)

SSHAgent creates an Agent that communicates over SSH. It returns non-nil error when connection could not be established.

type AggregationCA added in v1.14.3 

type AggregationCA struct{}

AggregationCA is a certificate authority for kubernetes aggregation API server

func (AggregationCA) IssueClientCertificate added in v1.14.3 

func (a AggregationCA) IssueClientCertificate(ctx context.Context, inf Infrastructure) (cert, key string, err error)

IssueClientCertificate issues TLS client certificate for API server

type BindPropagation 

type BindPropagation string

BindPropagation is bind propagation option for Docker https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation

func (BindPropagation) String 

func (p BindPropagation) String() string

type CNIConfFile added in v1.13.7 

type CNIConfFile struct {
	Name    string `json:"name"`
	Content string `json:"content"`
}

CNIConfFile is a config file for CNI plugin deployed on worker nodes by CKE.

type Cluster 

type Cluster struct {
	Name          string   `json:"name"`
	Nodes         []*Node  `json:"nodes"`
	TaintCP       bool     `json:"taint_control_plane"`
	CPTolerations []string `json:"control_plane_tolerations"`
	ServiceSubnet string   `json:"service_subnet"`
	DNSServers    []string `json:"dns_servers"`
	DNSService    string   `json:"dns_service"`
	Reboot        Reboot   `json:"reboot"`
	Options       Options  `json:"options"`
}

Cluster is a set of configurations for a etcd/Kubernetes cluster.

func NewCluster 

func NewCluster() *Cluster

NewCluster creates Cluster

func (*Cluster) Validate 

func (c *Cluster) Validate(isTmpl bool) error

Validate validates the cluster definition.

type ClusterDNSStatus 

type ClusterDNSStatus struct {
	ConfigMap *corev1.ConfigMap
	ClusterIP string
}

ClusterDNSStatus contains cluster resolver status.

type ClusterStatus 

type ClusterStatus struct {
	ConfigVersion string
	Name          string
	NodeStatuses  map[string]*NodeStatus // keys are IP address strings.

	Etcd       EtcdClusterStatus
	Kubernetes KubernetesClusterStatus
}

ClusterStatus represents the working cluster status. The structure reflects Cluster, of course.

type Command 

type Command struct {
	Name   string `json:"name"`
	Target string `json:"target"`
}

Command represents some command

func (Command) String 

func (c Command) String() string

String implements fmt.Stringer

type Commander 

type Commander interface {
	// Run executes the command
	Run(ctx context.Context, inf Infrastructure, leaderKey string) error
	// Command returns the command information
	Command() Command
}

Commander is a single step to proceed an operation

type Constraints 

type Constraints struct {
	ControlPlaneCount        int `json:"control-plane-count"`
	MinimumWorkers           int `json:"minimum-workers"`
	MaximumWorkers           int `json:"maximum-workers"`
	RebootMaximumUnreachable int `json:"maximum-unreachable-nodes-for-reboot"`
}

Constraints is a set of conditions that a cluster must satisfy

func DefaultConstraints 

func DefaultConstraints() *Constraints

DefaultConstraints returns the default constraints

func (*Constraints) Check 

func (c *Constraints) Check(cluster *Cluster) error

Check checks the cluster satisfies the constraints

type ContainerEngine 

type ContainerEngine interface {
	// PullImage pulls an image.
	PullImage(img Image) error
	// Run runs a container as a foreground process.
	Run(img Image, binds []Mount, command string, args ...string) error
	// RunWithInput runs a container as a foreground process with stdin as a string.
	RunWithInput(img Image, binds []Mount, command, input string, args ...string) error
	/// RunWithOutput runs a container as a foreground process and get stdout and stderr.
	RunWithOutput(img Image, binds []Mount, command string, args ...string) ([]byte, []byte, error)
	// RunSystem runs the named container as a system service.
	RunSystem(name string, img Image, opts []string, params, extra ServiceParams) error
	// Exists returns if named system container exists.
	Exists(name string) (bool, error)
	// Stop stops the named system container.
	Stop(name string) error
	// Kill kills the named system container.
	Kill(name string) error
	// Remove removes the named system container.
	Remove(name string) error
	// Inspect returns ServiceStatus for the named container.
	Inspect(name []string) (map[string]ServiceStatus, error)
	// VolumeCreate creates a local volume.
	VolumeCreate(name string) error
	// VolumeRemove creates a local volume.
	VolumeRemove(name string) error
	// VolumeExists returns true if the named volume exists.
	VolumeExists(name string) (bool, error)
}

ContainerEngine defines interfaces for a container engine.

func Docker 

func Docker(agent Agent) ContainerEngine

Docker is an implementation of ContainerEngine.

type EtcdCA 

type EtcdCA struct{}

EtcdCA is a certificate authority for etcd cluster.

func (EtcdCA) IssueForAPIServer 

func (e EtcdCA) IssueForAPIServer(ctx context.Context, inf Infrastructure, node *Node) (crt, key string, err error)

IssueForAPIServer issues TLC client certificate for Kubernetes.

func (EtcdCA) IssuePeerCert 

func (e EtcdCA) IssuePeerCert(ctx context.Context, inf Infrastructure, node *Node) (crt, key string, err error)

IssuePeerCert issues TLS certificates for mutual peer authentication.

func (EtcdCA) IssueRoot 

func (e EtcdCA) IssueRoot(ctx context.Context, inf Infrastructure) (cert, key string, err error)

IssueRoot issues certificate for root user.

func (EtcdCA) IssueServerCert 

func (e EtcdCA) IssueServerCert(ctx context.Context, inf Infrastructure, node *Node) (crt, key string, err error)

IssueServerCert issues TLS server certificates.

type EtcdClusterStatus 

type EtcdClusterStatus struct {
	IsHealthy     bool
	Members       map[string]*etcdserverpb.Member
	InSyncMembers map[string]bool
}

EtcdClusterStatus is the status of the etcd cluster.

type EtcdParams 

type EtcdParams struct {
	ServiceParams `json:",inline"`
	VolumeName    string `json:"volume_name"`
}

EtcdParams is a set of extra parameters for etcd.

type EtcdStatus 

type EtcdStatus struct {
	ServiceStatus
	HasData       bool
	IsAddedMember bool
}

EtcdStatus is the status of kubelet.

type Image 

type Image string

Image is the type of container images.

func (Image) Name 

func (i Image) Name() string

Name returns docker image name.

type InfoOperator added in v1.18.3 

type InfoOperator interface {
	Operator
	Info() string
}

InfoOperator is an extension of Operator that provides some information after the operation

type Infrastructure 

type Infrastructure interface {
	Close()

	// Agent returns the agent corresponding to addr and returns nil if addr is not connected.
	Agent(addr string) Agent
	Engine(addr string) ContainerEngine
	Vault() (*vault.Client, error)
	Storage() Storage

	NewEtcdClient(ctx context.Context, endpoints []string) (*clientv3.Client, error)
	K8sConfig(ctx context.Context, n *Node) (*rest.Config, error)
	K8sClient(ctx context.Context, n *Node) (*kubernetes.Clientset, error)
	HTTPClient() *well.HTTPClient
	HTTPSClient(ctx context.Context) (*well.HTTPClient, error)

	ReleaseAgent(addrs string)
}

Infrastructure presents an interface for infrastructure on CKE

func NewInfrastructure 

func NewInfrastructure(ctx context.Context, c *Cluster, s Storage) (Infrastructure, error)

NewInfrastructure creates a new Infrastructure instance

type IssueResponse 

type IssueResponse struct {
	Cert   string `json:"certificate"`
	Key    string `json:"private_key"`
	CACert string `json:"ca_certificate"`
}

IssueResponse is cli output format.

type KubeComponentStatus 

type KubeComponentStatus struct {
	ServiceStatus
	IsHealthy bool
}

KubeComponentStatus represents service status and endpoint's health

type KubeHTTP added in v1.19.6 

type KubeHTTP struct {
	// contains filtered or unexported fields
}

KubeHTTP provides TLS client certificate to access kube-apiserver. The certificate is cached in memory in order to avoid excessive certificate issuance.

func (*KubeHTTP) CACert added in v1.19.6 

func (k *KubeHTTP) CACert() string

CACert returns the CA certificate of kube-apiserver.

func (*KubeHTTP) Client added in v1.19.6 

func (k *KubeHTTP) Client() *well.HTTPClient

Client returns a HTTP client to acess kube-apiserver.

func (*KubeHTTP) GetCert added in v1.19.6 

func (k *KubeHTTP) GetCert(ctx context.Context, inf Infrastructure) (cert, key []byte, err error)

GetCert retrieves cached TLS client certificate to access kube-apiserver.

func (*KubeHTTP) Init added in v1.19.6 

func (k *KubeHTTP) Init(ctx context.Context, inf Infrastructure) error

Init initializes KubeHTTP.

type KubeletParams 

type KubeletParams struct {
	ServiceParams `json:",inline"`
	BootTaints    []corev1.Taint             `json:"boot_taints"`
	CNIConfFile   CNIConfFile                `json:"cni_conf_file"`
	Config        *unstructured.Unstructured `json:"config,omitempty"`
	CRIEndpoint   string                     `json:"cri_endpoint"`
}

KubeletParams is a set of extra parameters for kubelet.

func (KubeletParams) MergeConfig added in v1.19.0 

func (p KubeletParams) MergeConfig(base *kubeletv1beta1.KubeletConfiguration) (*kubeletv1beta1.KubeletConfiguration, error)

MergeConfig merges the input struct with `base`.

type KubeletStatus 

type KubeletStatus struct {
	ServiceStatus
	IsHealthy bool
	Config    *kubeletv1beta1.KubeletConfiguration
}

KubeletStatus represents kubelet status and health

type KubernetesCA 

type KubernetesCA struct{}

KubernetesCA is a certificate authority for k8s cluster.

func (KubernetesCA) IssueForAPIServer 

func (k KubernetesCA) IssueForAPIServer(ctx context.Context, inf Infrastructure, n *Node, serviceSubnet, clusterDomain string) (crt, key string, err error)

IssueForAPIServer issues TLS certificate for API servers.

func (KubernetesCA) IssueForControllerManager 

func (k KubernetesCA) IssueForControllerManager(ctx context.Context, inf Infrastructure) (crt, key string, err error)

IssueForControllerManager issues TLS certificate for kube-controller-manager.

func (KubernetesCA) IssueForKubelet 

func (k KubernetesCA) IssueForKubelet(ctx context.Context, inf Infrastructure, node *Node) (crt, key string, err error)

IssueForKubelet issues TLS certificate for kubelet.

func (KubernetesCA) IssueForProxy 

func (k KubernetesCA) IssueForProxy(ctx context.Context, inf Infrastructure) (crt, key string, err error)

IssueForProxy issues TLS certificate for kube-proxy.

func (KubernetesCA) IssueForScheduler 

func (k KubernetesCA) IssueForScheduler(ctx context.Context, inf Infrastructure) (crt, key string, err error)

IssueForScheduler issues TLS certificate for kube-scheduler.

func (KubernetesCA) IssueForServiceAccount 

func (k KubernetesCA) IssueForServiceAccount(ctx context.Context, inf Infrastructure) (crt, key string, err error)

IssueForServiceAccount issues TLS certificate to sign service account tokens.

func (KubernetesCA) IssueUserCert added in v1.14.14 

func (k KubernetesCA) IssueUserCert(ctx context.Context, inf Infrastructure, userName, groupName string, ttl string) (crt, key string, err error)

IssueUserCert issues client certificate for user.

type KubernetesClusterStatus 

type KubernetesClusterStatus struct {
	IsControlPlaneReady bool
	Nodes               []corev1.Node
	DNSService          *corev1.Service
	ClusterDNS          ClusterDNSStatus
	NodeDNS             NodeDNSStatus
	MasterEndpoints     *corev1.Endpoints
	MasterEndpointSlice *discoveryv1.EndpointSlice
	EtcdService         *corev1.Service
	EtcdEndpoints       *corev1.Endpoints
	EtcdEndpointSlice   *discoveryv1.EndpointSlice
	ResourceStatuses    map[string]ResourceStatus
}

KubernetesClusterStatus contains kubernetes cluster configurations

func (KubernetesClusterStatus) IsReady 

func (s KubernetesClusterStatus) IsReady(cluster *Cluster) bool

IsReady returns the cluster condition whether or not Pod can be scheduled

func (KubernetesClusterStatus) SetResourceStatus added in v1.13.9 

func (s KubernetesClusterStatus) SetResourceStatus(rkey string, ann map[string]string, isManaged bool)

SetResourceStatus sets status of the resource.

type Mount 

type Mount struct {
	Source      string          `json:"source"`
	Destination string          `json:"destination"`
	ReadOnly    bool            `json:"read_only"`
	Propagation BindPropagation `json:"propagation"`
	Label       SELinuxLabel    `json:"selinux_label"`
}

Mount is volume mount information

func (Mount) Equal 

func (m Mount) Equal(o Mount) bool

Equal returns true if the mount is equals to other one, otherwise return false

type Node 

type Node struct {
	Address      string            `json:"address"`
	Hostname     string            `json:"hostname"`
	User         string            `json:"user"`
	ControlPlane bool              `json:"control_plane"`
	Annotations  map[string]string `json:"annotations"`
	Labels       map[string]string `json:"labels"`
	Taints       []corev1.Taint    `json:"taints"`
}

Node represents a node in Kubernetes.

func ControlPlanes 

func ControlPlanes(nodes []*Node) []*Node

ControlPlanes returns control planes []*Node

func Workers added in v1.17.6 

func Workers(nodes []*Node) []*Node

Workers returns workers []*Node

func (*Node) Nodename 

func (n *Node) Nodename() string

Nodename returns a hostname or address if hostname is empty

type NodeDNSStatus 

type NodeDNSStatus struct {
	ConfigMap *corev1.ConfigMap
}

NodeDNSStatus contains node local resolver status.

type NodeStatus 

type NodeStatus struct {
	SSHConnected      bool
	Etcd              EtcdStatus
	Rivers            ServiceStatus
	EtcdRivers        ServiceStatus
	APIServer         KubeComponentStatus
	ControllerManager KubeComponentStatus
	Scheduler         SchedulerStatus
	Proxy             ProxyStatus
	Kubelet           KubeletStatus
	Labels            map[string]string // are labels for k8s Node resource.
}

NodeStatus status of a node.

type OperationPhase added in v1.16.1 

type OperationPhase string

OperationPhase represents the processing status of CKE server.

type Operator 

type Operator interface {
	// Name returns the operation name.
	Name() string
	// NextCommand returns the next command or nil if completed.
	NextCommand() Commander
	// Targets returns the ip which will be affected by the operation
	Targets() []string
}

Operator is the interface for operations

type Options 

type Options struct {
	Etcd              EtcdParams      `json:"etcd"`
	Rivers            ServiceParams   `json:"rivers"`
	EtcdRivers        ServiceParams   `json:"etcd-rivers"`
	APIServer         APIServerParams `json:"kube-api"`
	ControllerManager ServiceParams   `json:"kube-controller-manager"`
	Scheduler         SchedulerParams `json:"kube-scheduler"`
	Proxy             ProxyParams     `json:"kube-proxy"`
	Kubelet           KubeletParams   `json:"kubelet"`
}

Options is a set of optional parameters for k8s components.

type ProxyMode added in v1.19.2 

type ProxyMode string

ProxyMode is a type for kube-proxy's --proxy-mode argument.

type ProxyParams added in v1.19.2 

type ProxyParams struct {
	ServiceParams `json:",inline"`
	Disable       bool                       `json:"disable,omitempty"`
	Config        *unstructured.Unstructured `json:"config,omitempty"`
}

ProxyParams is a set of extra parameters for kube-proxy.

func (ProxyParams) GetMode added in v1.19.2 

func (p ProxyParams) GetMode() string

GetMode returns the proxy mode.

func (ProxyParams) MergeConfig added in v1.20.0 

func (p ProxyParams) MergeConfig(base *proxyv1alpha1.KubeProxyConfiguration) (*proxyv1alpha1.KubeProxyConfiguration, error)

MergeConfig merges the input struct with `base`.

type ProxyStatus added in v1.20.0 

type ProxyStatus struct {
	ServiceStatus
	IsHealthy bool
	Config    *proxyv1alpha1.KubeProxyConfiguration
}

ProxyStatus represents kubelet status and health

type Reboot added in v1.18.3 

type Reboot struct {
	RebootCommand          []string              `json:"reboot_command"`
	BootCheckCommand       []string              `json:"boot_check_command"`
	MaxConcurrentReboots   *int                  `json:"max_concurrent_reboots,omitempty"`
	EvictionTimeoutSeconds *int                  `json:"eviction_timeout_seconds,omitempty"`
	CommandTimeoutSeconds  *int                  `json:"command_timeout_seconds,omitempty"`
	CommandRetries         *int                  `json:"command_retries"`
	ProtectedNamespaces    *metav1.LabelSelector `json:"protected_namespaces,omitempty"`
}

Reboot is a set of configurations for reboot.

type RebootQueueEntry added in v1.18.3 

type RebootQueueEntry struct {
	Index              int64        `json:"index,string"`
	Node               string       `json:"node"`
	Status             RebootStatus `json:"status"`
	LastTransitionTime time.Time    `json:"last_transition_time,omitempty"`
	DrainBackOffCount  int          `json:"drain_backoff_count,omitempty"`
	DrainBackOffExpire time.Time    `json:"drain_backoff_expire,omitempty"`
}

RebootQueueEntry represents a queue entry of reboot operation

func DedupRebootQueueEntries added in v1.22.8 

func DedupRebootQueueEntries(entries []*RebootQueueEntry) []*RebootQueueEntry

func NewRebootQueueEntry added in v1.18.3 

func NewRebootQueueEntry(node string) *RebootQueueEntry

NewRebootQueueEntry creates new `RebootQueueEntry`. `Index` will be supplied in registration.

func (*RebootQueueEntry) ClusterMember added in v1.22.8 

func (entry *RebootQueueEntry) ClusterMember(c *Cluster) bool

ClusterMember returns whether the node in this entry is a cluster member.

type RebootStatus added in v1.18.3 

type RebootStatus string

RebootStatus is status of reboot operation

type Record 

type Record struct {
	ID        int64        `json:"id,string"`
	Status    RecordStatus `json:"status"`
	Operation string       `json:"operation"`
	Command   Command      `json:"command"`
	Targets   []string     `json:"targets"`
	Info      string       `json:"info"`
	Error     string       `json:"error"`
	StartAt   time.Time    `json:"start-at"`
	EndAt     time.Time    `json:"end-at"`
}

Record represents a record of an operation

func NewRecord 

func NewRecord(id int64, op string, targets []string) *Record

NewRecord creates new `Record`

func (*Record) Cancel 

func (r *Record) Cancel()

Cancel cancels the operation

func (*Record) Complete 

func (r *Record) Complete()

Complete completes the operation

func (*Record) SetCommand 

func (r *Record) SetCommand(c Command)

SetCommand updates the record for the new command

func (*Record) SetError 

func (r *Record) SetError(e error)

SetError cancels the operation with error information

func (*Record) SetInfo added in v1.18.3 

func (r *Record) SetInfo(i string)

SetInfo records some information of the operation result

type RecordChan added in v1.14.1 

type RecordChan <-chan *Record

RecordChan is a channel for watching new operation records.

type RecordStatus 

type RecordStatus string

RecordStatus is status of an operation

type ResourceDefinition added in v1.13.9 

type ResourceDefinition struct {
	Key        string
	Kind       string
	Namespace  string
	Name       string
	Revision   int64
	Image      string // may contains multiple images; we should not use this whole string as an image name.
	Definition []byte
}

ResourceDefinition represents a CKE-managed kubernetes resource.

func (ResourceDefinition) NeedUpdate added in v1.13.10 

func (d ResourceDefinition) NeedUpdate(rs *ResourceStatus) bool

NeedUpdate returns true if annotations of the current resource indicates need for update.

func (ResourceDefinition) String added in v1.13.9 

func (d ResourceDefinition) String() string

String implements fmt.Stringer.

type ResourceStatus added in v1.17.0 

type ResourceStatus struct {
	// Annotations is the copy of metadata.annotations
	Annotations map[string]string
	// HasBeenSSA indicates that this resource has been already updated by server-side apply
	HasBeenSSA bool
}

ResourceStatus represents the status of registered K8s resources

type SELinuxLabel 

type SELinuxLabel string

SELinuxLabel is selinux label of the host file or directory https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label

func (SELinuxLabel) String 

func (l SELinuxLabel) String() string

type SchedulerParams added in v1.14.6 

type SchedulerParams struct {
	ServiceParams `json:",inline"`
	Config        *unstructured.Unstructured `json:"config,omitempty"`
}

SchedulerParams is a set of extra parameters for kube-scheduler.

func (SchedulerParams) MergeConfig added in v1.19.0 

func (p SchedulerParams) MergeConfig(base *schedulerv1beta3.KubeSchedulerConfiguration) (*schedulerv1beta3.KubeSchedulerConfiguration, error)

MergeConfig merges the input struct `base`.

type SchedulerStatus added in v1.14.6 

type SchedulerStatus struct {
	ServiceStatus
	IsHealthy bool
	Config    *schedulerv1beta3.KubeSchedulerConfiguration
}

SchedulerStatus represents kube-scheduler status and health

type ServerStatus added in v1.16.1 

type ServerStatus struct {
	Phase     OperationPhase `json:"phase"`
	Timestamp time.Time      `json:"timestamp"`
}

ServerStatus represents the current server status.

type ServiceParams 

type ServiceParams struct {
	ExtraArguments []string          `json:"extra_args"`
	ExtraBinds     []Mount           `json:"extra_binds"`
	ExtraEnvvar    map[string]string `json:"extra_env"`
}

ServiceParams is a common set of extra parameters for k8s components.

func (ServiceParams) Equal 

func (s ServiceParams) Equal(o ServiceParams) bool

Equal returns true if the services params is equals to other one, otherwise return false

type ServiceStatus 

type ServiceStatus struct {
	Running       bool
	Image         string
	BuiltInParams ServiceParams
	ExtraParams   ServiceParams
}

ServiceStatus represents statuses of a service.

If Running is false, the service is not running on the node. ExtraXX are extra parameters of the running service, if any.

type Storage 

type Storage struct {
	*clientv3.Client
}

Storage provides operations to store/retrieve CKE data in etcd.

func (Storage) DeleteRebootsEntry added in v1.18.3 

func (s Storage) DeleteRebootsEntry(ctx context.Context, leaderKey string, index int64) error

DeleteRebootsEntry deletes the entry specified by the index from the reboot queue.

func (Storage) DeleteResource added in v1.13.9 

func (s Storage) DeleteResource(ctx context.Context, key string) error

DeleteResource removes a user resource from etcd.

func (Storage) EnableRebootQueue added in v1.18.7 

func (s Storage) EnableRebootQueue(ctx context.Context, flag bool) error

EnableRebootQueue enables reboot queue processing when flag is true. When flag is false, reboot queue is not processed.

func (Storage) EnableSabakan added in v1.14.5 

func (s Storage) EnableSabakan(ctx context.Context, flag bool) error

EnableSabakan enables sabakan integration when flag is true. When flag is false, sabakan integration is disabled.

func (Storage) GetAllResources added in v1.13.9 

func (s Storage) GetAllResources(ctx context.Context) ([]ResourceDefinition, error)

GetAllResources gets all user-defined resources. The returned slice of resources are sorted so that creating resources in order will not fail.

func (Storage) GetCACertificate 

func (s Storage) GetCACertificate(ctx context.Context, name string) (string, error)

GetCACertificate loads CA certificate from etcd.

func (Storage) GetCluster 

func (s Storage) GetCluster(ctx context.Context) (*Cluster, error)

GetCluster loads *Cluster from etcd. If cluster configuration has not been stored, this returns ErrNotFound.

func (Storage) GetClusterWithRevision 

func (s Storage) GetClusterWithRevision(ctx context.Context) (*Cluster, int64, error)

GetClusterWithRevision loads *Cluster from etcd as well as the stored revision number. The revision number was stored with *Cluster by PutClusterWithTemplateRevision().

func (Storage) GetConfigVersion added in v1.15.7 

func (s Storage) GetConfigVersion(ctx context.Context) (string, error)

GetConfigVersion retrieves the configuration version of the Kubernetes cluster.

func (Storage) GetConstraints 

func (s Storage) GetConstraints(ctx context.Context) (*Constraints, error)

GetConstraints loads *Constraints from etcd. If constraints have not been stored, this returns ErrNotFound.

func (Storage) GetLeaderHostname 

func (s Storage) GetLeaderHostname(ctx context.Context) (string, error)

GetLeaderHostname returns the current leader's host name. It returns non-nil error when there is no leader.

func (Storage) GetRebootsEntries added in v1.18.3 

func (s Storage) GetRebootsEntries(ctx context.Context) ([]*RebootQueueEntry, error)

GetRebootsEntries loads the entries from the reboot queue.

func (Storage) GetRebootsEntry added in v1.18.3 

func (s Storage) GetRebootsEntry(ctx context.Context, index int64) (*RebootQueueEntry, error)

GetRebootsEntry loads the entry specified by the index from the reboot queue. If the pointed entry is not found, this returns ErrNotFound.

func (Storage) GetRecords 

func (s Storage) GetRecords(ctx context.Context, count int64) ([]*Record, error)

GetRecords loads list of *Record from etcd. The returned records are sorted by record ID in decreasing order.

func (Storage) GetResource added in v1.13.9 

func (s Storage) GetResource(ctx context.Context, key string) ([]byte, int64, error)

GetResource gets a user resource.

func (Storage) GetSabakanQueryVariables 

func (s Storage) GetSabakanQueryVariables(ctx context.Context) ([]byte, error)

GetSabakanQueryVariables gets query variables for Sabakan.

func (Storage) GetSabakanTemplate 

func (s Storage) GetSabakanTemplate(ctx context.Context) (*Cluster, int64, error)

GetSabakanTemplate gets template cluster configuration. If a template exists, it will be returned with ModRevision.

func (Storage) GetSabakanURL 

func (s Storage) GetSabakanURL(ctx context.Context) (string, error)

GetSabakanURL gets URL of sabakan API. The URL must be an absolute URL pointing GraphQL endpoint.

func (Storage) GetServiceAccountCert 

func (s Storage) GetServiceAccountCert(ctx context.Context) (string, error)

GetServiceAccountCert loads x509 certificate for service account. The format is PEM.

func (Storage) GetServiceAccountKey 

func (s Storage) GetServiceAccountKey(ctx context.Context) (string, error)

GetServiceAccountKey loads private key for service account. The format is PEM.

func (Storage) GetStatus added in v1.16.1 

func (s Storage) GetStatus(ctx context.Context) (*ServerStatus, error)

GetStatus retrieves the server status if exists. If the status is not found, this returns ("", ErrNotFound).

func (Storage) GetVaultConfig 

func (s Storage) GetVaultConfig(ctx context.Context) (*VaultConfig, error)

GetVaultConfig loads *VaultConfig from etcd.

func (Storage) IsRebootQueueDisabled added in v1.18.7 

func (s Storage) IsRebootQueueDisabled(ctx context.Context) (bool, error)

IsRebootQueueDisabled returns true if reboot queue is disabled.

func (Storage) IsSabakanDisabled added in v1.14.5 

func (s Storage) IsSabakanDisabled(ctx context.Context) (bool, error)

IsSabakanDisabled returns true if sabakan integration is disabled.

func (Storage) ListResources added in v1.13.9 

func (s Storage) ListResources(ctx context.Context) ([]string, error)

ListResources lists keys of registered user resources.

func (Storage) NextRecordID 

func (s Storage) NextRecordID(ctx context.Context) (int64, error)

NextRecordID get the next record ID from etcd

func (Storage) PutCACertificate 

func (s Storage) PutCACertificate(ctx context.Context, name, pem string) error

PutCACertificate stores CA certificate into etcd.

func (Storage) PutCluster 

func (s Storage) PutCluster(ctx context.Context, c *Cluster) error

PutCluster stores *Cluster into etcd.

func (Storage) PutClusterWithTemplateRevision 

func (s Storage) PutClusterWithTemplateRevision(ctx context.Context, c *Cluster, rev int64, leaderKey string) error

PutClusterWithTemplateRevision stores *Cluster into etcd along with a revision number.

func (Storage) PutConfigVersion added in v1.15.7 

func (s Storage) PutConfigVersion(ctx context.Context, leaderKey string) error

PutConfigVersion sets the current configuration version of the Kubernetes cluster.

func (Storage) PutConstraints 

func (s Storage) PutConstraints(ctx context.Context, c *Constraints) error

PutConstraints stores *Constraints into etcd.

func (Storage) PutServiceAccountData 

func (s Storage) PutServiceAccountData(ctx context.Context, leaderKey, cert, key string) error

PutServiceAccountData stores x509 certificate and private key for service account.

func (Storage) PutVaultConfig 

func (s Storage) PutVaultConfig(ctx context.Context, c *VaultConfig) error

PutVaultConfig stores *VaultConfig into etcd.

func (Storage) RegisterRebootsEntry added in v1.18.3 

func (s Storage) RegisterRebootsEntry(ctx context.Context, r *RebootQueueEntry) error

RegisterRebootsEntry enqueues a reboot queue entry to the reboot queue. "Index" of the entry is retrieved and updated in this method. The given value is ignored.

func (Storage) RegisterRecord 

func (s Storage) RegisterRecord(ctx context.Context, leaderKey string, r *Record) error

RegisterRecord stores *Record if the leaderKey exists

func (Storage) SetResource added in v1.13.9 

func (s Storage) SetResource(ctx context.Context, key, value string) error

SetResource sets a user resource.

func (Storage) SetSabakanQueryVariables 

func (s Storage) SetSabakanQueryVariables(ctx context.Context, vars string) error

SetSabakanQueryVariables sets query variables for Sabakan. Caller must validate the contents.

func (Storage) SetSabakanTemplate 

func (s Storage) SetSabakanTemplate(ctx context.Context, tmpl *Cluster) error

SetSabakanTemplate stores template cluster configuration. Caller must validate the template.

func (Storage) SetSabakanURL 

func (s Storage) SetSabakanURL(ctx context.Context, url string) error

SetSabakanURL stores URL of sabakan API.

func (Storage) SetStatus added in v1.16.1 

func (s Storage) SetStatus(ctx context.Context, lease clientv3.LeaseID, st *ServerStatus) error

SetStatus stores the server status.

func (Storage) UpdateRebootsEntry added in v1.18.3 

func (s Storage) UpdateRebootsEntry(ctx context.Context, r *RebootQueueEntry) error

UpdateRebootsEntry updates existing reboot queue entry. It always overwrites the contents with a CAS loop. If the entry is not found in the reboot queue, this returns ErrNotFound.

func (Storage) UpdateRecord 

func (s Storage) UpdateRecord(ctx context.Context, leaderKey string, r *Record) error

UpdateRecord updates existing record

func (Storage) WatchRecords added in v1.14.1 

func (s Storage) WatchRecords(ctx context.Context, initialCount int64) (RecordChan, error)

WatchRecords watches new operation records. The watched records will be returned through the returned channel.

type VaultConfig 

type VaultConfig struct {
	// Endpoint is the address of the Vault server.
	Endpoint string `json:"endpoint"`

	// CACert is x509 certificate in PEM format of the endpoint CA.
	CACert string `json:"ca-cert"`

	// RoleID is AppRole ID to login to Vault.
	RoleID string `json:"role-id"`

	// SecretID is AppRole secret to login to Vault.
	SecretID string `json:"secret-id"`
}

VaultConfig is data to store in etcd

func (*VaultConfig) Validate 

func (c *VaultConfig) Validate() error

Validate validates the vault configuration

type WebhookCA added in v1.18.1 

type WebhookCA struct{}

WebhookCA is a certificate authority for kubernetes admission webhooks

func (WebhookCA) IssueCertificate added in v1.18.1 

func (WebhookCA) IssueCertificate(ctx context.Context, inf Infrastructure, namespace, name string) (cert, key string, err error)

IssueCertificate issues TLS server certificate `namespace` and `name` specifies the namespace/name of a webhook Service.

Source Files

View all Source files

Directories

Path Synopsis
op
clusterdns
common
Package common provides generic commands shared by many Operators.
 Package common provides generic commands shared by many Operators.
etcd
k8s
nodedns
pkg
cke
cke-localproxy
ckecli
ckecli/cmd
compile_resources
mock
tools
empty-dir
install-cni
make_directories
rivers
write_files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.