How to use IPVS

This document shows how to use kube-proxy ipvs mode.

What is IPVS

IPVS (IP Virtual Server) implements transport-layer load balancing, usually called Layer 4 LAN switching, as part of Linux kernel.

IPVS runs on a host and acts as a load balancer in front of a cluster of real servers. IPVS can direct requests for TCP and UDP-based services to the real servers, and make services of real servers appear as virtual services on a single IP address.

Run kube-proxy in ipvs mode

Currently, local-up scripts and kubeadm support switching IPVS proxy mode via exporting environment variables or specifying flags.

Local UP Cluster

Kube-proxy will run in iptables mode by default in a local-up cluster.

Users should export the env KUBE_PROXY_MODE=ipvs to specify the ipvs mode before deploying the cluster if want to run kube-proxy in ipvs mode.

Cluster Created by Kubeadm

Kube-proxy will run in iptables mode by default in a cluster deployed by kubeadm.

If you are using kubeadm with a configuration file, you can specify the ipvs mode adding SupportIPVSProxyMode: true below the kubeProxy field. Then the configuration file is similar to:

kind: MasterConfiguration
    featureGates: SupportIPVSProxyMode=true
    mode: ipvs


Check IPVS proxy rules

People can use ipvsadm tool to check whether kube-proxy are maintaining IPVS rules correctly. For example, we may get IPVS proxy rules like:

# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP rr persistent 10800
  ->             Masq    1      0          0         
TCP rr      
UDP rr
Why kube-proxy can't start IPVS mode

People can do the following check list step by step:

1. Enable IPVS feature gateway

Currently IPVS-based kube-proxy is still in alpha phase, people need to enable --feature-gates=SupportIPVSProxyMode=true explicitly.

2. Specify proxy-mode=ipvs

Tell kube-proxy that proxy-mode=ipvs, please.

3. Load ipvs required kernel modules

The following kernel modules are required by IPVS-based kube-proxy:


IPVS-based kube-proxy will load them automatically. If it fails to load them, please check whether they are compiled into your kernel.

Expand ▾ Collapse ▴




View Source
const (
	// MinIPSetCheckVersion is the min ipset version we need.  IPv6 is supported in ipset 6.x
	MinIPSetCheckVersion = "6.0"

	// KubeLoopBackIPSet is used to store endpoints dst ip:port, source ip for solving hairpin purpose.
	KubeLoopBackIPSet = "KUBE-LOOP-BACK"

	// KubeClusterIPSet is used to store service cluster ip + port for masquerade purpose.
	KubeClusterIPSet = "KUBE-CLUSTER-IP"

	// KubeExternalIPSet is used to store service external ip + port for masquerade and filter purpose.
	KubeExternalIPSet = "KUBE-EXTERNAL-IP"

	// KubeLoadBalancerSet is used to store service load balancer ingress ip + port, it is the service lb portal.
	KubeLoadBalancerSet = "KUBE-LOAD-BALANCER"

	// KubeLoadBalancerMasqSet is used to store service load balancer ingress ip + port for masquerade purpose.
	KubeLoadBalancerMasqSet = "KUBE-LOAD-BALANCER-MASQ"

	// KubeLoadBalancerSourceIPSet is used to store service load balancer ingress ip + port + source IP for packet filter purpose.

	// KubeLoadBalancerSourceCIDRSet is used to store service load balancer ingress ip + port + source cidr for packet filter purpose.

	// KubeNodePortSetTCP is used to store nodeport TCP port for masquerade purpose.

	// KubeNodePortSetUDP is used to store nodeport UDP port for masquerade purpose.
View Source
const (

	// KubeServiceIPSetsChain is the services access IP chain
	KubeServiceIPSetsChain utiliptables.Chain = "KUBE-SVC-IPSETS"

	// KubeFireWallChain is the kubernetes firewall chain.
	KubeFireWallChain utiliptables.Chain = "KUBE-FIRE-WALL"

	// KubeMarkMasqChain is the mark-for-masquerade chain
	KubeMarkMasqChain utiliptables.Chain = "KUBE-MARK-MASQ"

	// KubeMarkDropChain is the mark-for-drop chain
	KubeMarkDropChain utiliptables.Chain = "KUBE-MARK-DROP"

	// DefaultScheduler is the default ipvs scheduler algorithm - round robin.
	DefaultScheduler = "rr"

	// DefaultDummyDevice is the default dummy interface where ipvs service address will bind to it.
	DefaultDummyDevice = "kube-ipvs0"
View Source
const EntryInvalidErr = "error adding entry %s to ipset %s"

    EntryInvalidErr indicates if an ipset entry is invalid or not


    This section is empty.


    func CanUseIPVSProxier

    func CanUseIPVSProxier(handle KernelHandler, ipsetver IPSetVersioner) (bool, error)

      CanUseIPVSProxier returns true if we can use the ipvs Proxier. This is determined by checking if all the required kernel modules can be loaded. It may return an error if it fails to get the kernel modules information without error, in which case it will also return false.

      func CleanupLeftovers

      func CleanupLeftovers(ipvs utilipvs.Interface, ipt utiliptables.Interface, ipset utilipset.Interface, cleanupIPVS bool) (encounteredError bool)

        CleanupLeftovers clean up all ipvs and iptables rules created by ipvs Proxier.


        type IPGetter

        type IPGetter interface {
        	NodeIPs() ([]net.IP, error)

          IPGetter helps get node network interface IP

          type IPSet

          type IPSet struct {
          	// contains filtered or unexported fields

            IPSet wraps util/ipset which is used by IPVS proxier.

            func NewIPSet

            func NewIPSet(handle utilipset.Interface, name string, setType utilipset.Type, isIPv6 bool) *IPSet

              NewIPSet initialize a new IPSet struct

              type IPSetVersioner

              type IPSetVersioner interface {
              	// returns "X.Y"
              	GetVersion() (string, error)

                IPSetVersioner can query the current ipset version.

                type KernelHandler

                type KernelHandler interface {
                	GetModules() ([]string, error)

                  KernelHandler can handle the current installed kernel modules.

                  type LinuxKernelHandler

                  type LinuxKernelHandler struct {
                  	// contains filtered or unexported fields

                    LinuxKernelHandler implements KernelHandler interface.

                    func NewLinuxKernelHandler

                    func NewLinuxKernelHandler() *LinuxKernelHandler

                      NewLinuxKernelHandler initializes LinuxKernelHandler with exec.

                      func (*LinuxKernelHandler) GetModules

                      func (handle *LinuxKernelHandler) GetModules() ([]string, error)

                        GetModules returns all installed kernel modules.

                        type NetLinkHandle

                        type NetLinkHandle interface {
                        	// EnsureAddressBind checks if address is bound to the interface and, if not, binds it.  If the address is already bound, return true.
                        	EnsureAddressBind(address, devName string) (exist bool, err error)
                        	// UnbindAddress unbind address from the interface
                        	UnbindAddress(address, devName string) error
                        	// EnsureDummyDevice checks if dummy device is exist and, if not, create one.  If the dummy device is already exist, return true.
                        	EnsureDummyDevice(devName string) (exist bool, err error)
                        	// DeleteDummyDevice deletes the given dummy device by name.
                        	DeleteDummyDevice(devName string) error
                        	// GetLocalAddresses returns all unique local type IP addresses based on filter device interface.  If filter device is not given,
                        	// it will list all unique local type addresses.
                        	GetLocalAddresses(filterDev string) (sets.String, error)

                          NetLinkHandle for revoke netlink interface

                          func NewNetLinkHandle

                          func NewNetLinkHandle() NetLinkHandle

                            NewNetLinkHandle will crate a new NetLinkHandle

                            type Proxier

                            type Proxier struct {
                            	// contains filtered or unexported fields

                              Proxier is an ipvs based proxy for connections between a localhost:lport and services that provide the actual backends.

                              func NewProxier

                              func NewProxier(ipt utiliptables.Interface,
                              	ipvs utilipvs.Interface,
                              	ipset utilipset.Interface,
                              	sysctl utilsysctl.Interface,
                              	exec utilexec.Interface,
                              	syncPeriod time.Duration,
                              	minSyncPeriod time.Duration,
                              	masqueradeAll bool,
                              	masqueradeBit int,
                              	clusterCIDR string,
                              	hostname string,
                              	nodeIP net.IP,
                              	recorder record.EventRecorder,
                              	healthzServer healthcheck.HealthzUpdater,
                              	scheduler string,
                              	nodePortAddresses []string,
                              ) (*Proxier, error)

                                NewProxier returns a new Proxier given an iptables and ipvs Interface instance. Because of the iptables and ipvs logic, it is assumed that there is only a single Proxier active on a machine. An error will be returned if it fails to update or acquire the initial lock. Once a proxier is created, it will keep iptables and ipvs rules up to date in the background and will not terminate if a particular iptables or ipvs call fails.

                                func (*Proxier) OnEndpointsAdd

                                func (proxier *Proxier) OnEndpointsAdd(endpoints *api.Endpoints)

                                  OnEndpointsAdd is called whenever creation of new endpoints object is observed.

                                  func (*Proxier) OnEndpointsDelete

                                  func (proxier *Proxier) OnEndpointsDelete(endpoints *api.Endpoints)

                                    OnEndpointsDelete is called whenever deletion of an existing endpoints object is observed.

                                    func (*Proxier) OnEndpointsSynced

                                    func (proxier *Proxier) OnEndpointsSynced()

                                      OnEndpointsSynced is called once all the initial event handlers were called and the state is fully propagated to local cache.

                                      func (*Proxier) OnEndpointsUpdate

                                      func (proxier *Proxier) OnEndpointsUpdate(oldEndpoints, endpoints *api.Endpoints)

                                        OnEndpointsUpdate is called whenever modification of an existing endpoints object is observed.

                                        func (*Proxier) OnServiceAdd

                                        func (proxier *Proxier) OnServiceAdd(service *api.Service)

                                          OnServiceAdd is called whenever creation of new service object is observed.

                                          func (*Proxier) OnServiceDelete

                                          func (proxier *Proxier) OnServiceDelete(service *api.Service)

                                            OnServiceDelete is called whenever deletion of an existing service object is observed.

                                            func (*Proxier) OnServiceSynced

                                            func (proxier *Proxier) OnServiceSynced()

                                              OnServiceSynced is called once all the initial even handlers were called and the state is fully propagated to local cache.

                                              func (*Proxier) OnServiceUpdate

                                              func (proxier *Proxier) OnServiceUpdate(oldService, service *api.Service)

                                                OnServiceUpdate is called whenever modification of an existing service object is observed.

                                                func (*Proxier) Sync

                                                func (proxier *Proxier) Sync()

                                                  Sync is called to synchronize the proxier state to iptables and ipvs as soon as possible.

                                                  func (*Proxier) SyncLoop

                                                  func (proxier *Proxier) SyncLoop()

                                                    SyncLoop runs periodic work. This is expected to run as a goroutine or as the main loop of the app. It does not return.


                                                    Path Synopsis