package module
v0.0.0-...-1eec387 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: May 25, 2022 License: Apache-2.0 Imports: 30 Imported by: 0


GitHub release CircleCI GoDoc Go Report Card

neco logo

Neco is a project to build and maintain highly automated on-premise data centers using Kubernetes at the center of its system. The project is led by a CNCF Silver member Cybozu which is also known as Kintone in the United States.

Neco is also the name of the architecture for systems built on the deliverables of the project.

Please read about Project Neco: Building Cloud Native On-Premise Data Centers.

Table of Contents:




The project consists of a lot of repositories including:

System overview


A Neco data center consists of a few (from 3 to 5) management servers called boot sever and thousands of servers for a Kubernetes cluster; pretty big, huh? In fact, Neco is designed to manage such large clusters.

Boot servers

Boot servers are symmetrical with each other except for host names and IP addresses.

All persistent data are stored in an etcd cluster. Thanks to etcd, adding or removing a boot server can be done easily.

Sabakan is a network boot server that supports DHCP and HTTP. It can also manage an inventory of Node servers and assign IP addresses automatically. Node servers should use UEFI HTTP boot to load Flatcar from sabakan.

CKE, or Cybozu Kubernetes Engines, is a CNCF certified Kubernetes installer. It queries Node server status to Sabakan and chooses Nodes to construct Kubernetes control plane and worker nodes. TLS certificates required for Kubernetes are issued by Vault.

Boot servers are set up using a custom Ubuntu installer. installer directory contains the tools to build the custom installer.

Neco Continuous Delivery System (Neco CD)

To bootstrap and maintain boot servers, a simple continuous delivery system called Neco CD is implemented in this repository.

Neco CD consists of these programs and artifacts:

  • neco-updater: a daemon to watch GitHub and notify neco-worker of new artifacts.
  • neco-worker: a daemon to install/update programs on a boot server.
  • neco: CLI tool to bootstrap and configure boot servers.
  • Neco debian package: archive of program binaries to be installed on boot servers.

Debian packages are built by CircleCI and released on GitHub releases.

Read docs/ for details.


A Kubernetes cluster created with CKE is a vanilla installation; it almost has nothing useful.

We have selected a set of applications such as MetalLB, Calico, or Teleport to make the vanilla Kubernetes fully featured. The manifests of the applications are maintained in neco-apps repository and continuously delivered by Argo CD.



Network of Neco is designed with these goals:

  • Bare-metal performance, i.e., no overlays like VXLAN
  • Scalability, especially for east-west traffic
  • Vendor neutrality

The data center network for Neco should employ leaf-spine topology. More specifically, each data center rack works as an autonomous system (AS) and exchanges routes with BGP.

Since BGP is a layer-3 routing protocol, use of layer-2 can be limited within a single rack.


Node BGP and redundancy

Kubernetes nodes need to route packets for Pods. Without overlay networking, this means that each node needs to advertise routing information. In Neco, each node runs BIRD to speak BGP and advertise routes for Pods.

For high availability, network links must be redundant. Therefore, Neco requires that each rack has two Top-of-Rack (ToR) switches and each server has two ethernet links. To reduce convergence time of BGP, BFD is used together.

This means that ToR switches and node servers in a rack are all BGP speakers and need to connect each other as iBGP peers. To make configurations simple and flexible, ToR switches are configured as route reflectors.

The two ToR switches are independent; each has its own managing subnet. As a result, each server has two IP addresses for each ToR switch. To tolerate single link or ToR switch failure, the server has another IP address called the node address with /32 netmask and advertises it via BGP.


DHCP and location of boot servers

UEFI HTTP Boot requires a DHCP server which requires layer-2 broadcast domain. Without DHCP relay servers, a dedicated server must be run as a boot server for each rack because layer-2 domain does not span racks.

If ToR switches can provide DHCP relay, this requirement can be relaxed.


Virtual data center

To make continuous delivery reliable enough, thorough and comprehensive system tests need to be performed. Since Neco is the system to bootstrap and maintain an on-premise data center, this means that the tests should do the same thing, that is, to bootstrap and maintain a data center!

To simulate an on-premise data center, we have created placemat, a tool to construct a virtual data center using containers, virtual machines and Linux network stacks.

The virtual data center implements the aforementioned leaf-spine networks with BGP routers. To create the virtual data center, run the following commands on Ubuntu 20.04:

$ sudo add-apt-repository ppa:smoser/swtpm
$ echo 'deb /' | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
$ curl -fsSL | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/devel_kubic_libcontainers_stable.gpg > /dev/null
$ wget -O - | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] focal stable"
$ sudo apt update
$ sudo apt install -y build-essential systemd-container lldpd qemu qemu-kvm socat picocom swtpm cloud-utils bird2 squid chrony dnsmasq jq freeipmi-tools unzip skopeo fakeroot docker-ce docker-ce-cli
$ wget
$ wget
$ sudo install -m 0644 -b bios.bin bios-256k.bin /usr/share/seabios/
$ wget
$ sudo dpkg -i ./placemat2_2.2.0_amd64.deb
$ git clone
$ cd neco/dctest
$ make setup
$ make placemat
$ make test
  • To login a boot server in the virtual data center, run ./dcssh boot-0.
  • To stop and delete the virtual data center, run make stop.

The setup commands above are examined on a GCP VM based on the Ubuntu 20.04 disk image. Some more setup steps are needed for the GCP environment.

Large tests

dctest directory contains test suites to run large tests using the virtual data center. Details are described in docs/

Middle/Small tests

Other than large tests, this repository contains smaller tests that can be run as follows:

  1. Setup the environment by make setup. This is a one shot job.
  2. Run etcd by make start-etcd
  3. Run tests by go test -v -count 1 -race ./...
  4. Stop etcd by make stop-etcd


Other than Go packages, this repository has the following directories:

  • dctest/: Large tests using a virtual data center.
  • debian/: Ingredients for Debian package.
  • docs/: miscellaneous documentation.
  • etc/: Kubernetes manifests for networking and other configuration files.
  • ignitions/: Ignition template files.




View Source
const (
	NecoDir     = "/etc/neco"
	NecoDataDir = "/usr/share/neco"

	// NecoPrefix is the etcd key prefix for Neco tools.
	NecoPrefix = "/neco/"

	NecoPackageName = "neco"
	NecoUserAgent   = ""

Neco params

View Source
const (
	GitHubRepoOwner = "cybozu-go"
	GitHubRepoName  = "neco"

Neco repository

View Source
const (
	EtcdDir       = "/etc/etcd"
	EtcdUID       = 10000
	EtcdGID       = 10000
	EtcdDataDir   = "/var/lib/etcd-container"
	EtcdBackupDir = "/var/lib/etcd-backup"
	EtcdService   = "etcd-container"

Etcd params

View Source
const (
	VaultDir     = "/etc/vault"
	VaultUID     = 10000
	VaultGID     = 10000
	CAServer     = "ca/server"
	CAEtcdPeer   = "ca/boot-etcd-peer"
	CAEtcdClient = "ca/boot-etcd-client"
	TTL100Year   = "876000h"
	TTL10Year    = "87600h"
	VaultService = "vault"

	// VaultPrefix is the etcd key prefix for vault.
	VaultPrefix = "/vault/"

Vault params

View Source
const (
	EtcdpasswdDir = "/etc/etcdpasswd"

	EtcdpasswdService = "ep-agent"
	EtcdpasswdPrefix  = "/passwd/"

Etcdpasswd params

View Source
const (
	TeleportDir = "/etc/teleport"

	TeleportService = "teleport-node"

Teleport params

View Source
const (
	SabakanDir = "/etc/sabakan"

	SabakanService       = "sabakan"
	SabakanPrefix        = "/sabakan/"
	SabakanDataDir       = "/var/lib/sabakan"
	SabakanLocalEndpoint = ""

Sabakan params

View Source
const (
	SerfService = "serf"
	SerfDir     = "/etc/serf"

Serf params

View Source
const (
	CKEDir = "/etc/cke"

	CKEService = "cke"
	CKEPrefix  = "/cke/"

	CKELocalProxyService = "cke-localproxy"

CKE params

View Source
const (
	PromtailDir = "/etc/promtail"

	PromtailService = "promtail"

Promtail params

View Source
const (
	NoneEnv    = "none"
	TestEnv    = "test"
	StagingEnv = "staging"
	ProdEnv    = "prod"

Environments to use release or pre-release neco

View Source
const (
	CondNotRunning = iota

Possible update conditions.

View Source
const (
	BashCompletionDir = "/etc/bash_completion.d"

Bash completion

View Source
const (
	SabakanStateSetterService = "sabakan-state-setter"

SabakanStateSetter params

View Source
const (
	SetupHWService = "setup-hw"

setup-hw params

View Source
const (
	WorkerAssetsPath = "/usr/libexec/neco"

Assets path for worker node


View Source
var (
	RackFile        = filepath.Join(NecoDir, "rack")
	ClusterFile     = filepath.Join(NecoDir, "cluster")
	SabakanIPAMFile = filepath.Join(NecoDir, "sabakan_ipam.json")
	DCTestFile      = filepath.Join(NecoDir, "dctest")

	ServerCAFile   = "/usr/local/share/ca-certificates/neco.crt"
	ServerCertFile = filepath.Join(NecoDir, "server.crt")
	ServerKeyFile  = filepath.Join(NecoDir, "server.key")

	EtcdPeerCAFile   = filepath.Join(EtcdDir, "ca-peer.crt")
	EtcdClientCAFile = filepath.Join(EtcdDir, "ca-client.crt")
	EtcdPeerCertFile = filepath.Join(EtcdDir, "peer.crt")
	EtcdPeerKeyFile  = filepath.Join(EtcdDir, "peer.key")
	EtcdConfFile     = filepath.Join(EtcdDir, "etcd.conf.yml")

	EtcdBackupCertFile = filepath.Join(EtcdDir, "backup.crt")
	EtcdBackupKeyFile  = filepath.Join(EtcdDir, "backup.key")

	VaultCertFile = filepath.Join(VaultDir, "etcd.crt")
	VaultKeyFile  = filepath.Join(VaultDir, "etcd.key")
	VaultConfFile = filepath.Join(VaultDir, "config.hcl")

	EtcdpasswdCertFile = filepath.Join(EtcdpasswdDir, "etcd.crt")
	EtcdpasswdKeyFile  = filepath.Join(EtcdpasswdDir, "etcd.key")
	EtcdpasswdConfFile = filepath.Join(EtcdpasswdDir, "config.yml")
	EtcdpasswdDropIn   = "/etc/systemd/system/ep-agent.service.d/10-check-certificate.conf"

	TeleportConfFile = filepath.Join(TeleportDir, "teleport.yaml")

	SabakanCertFile           = filepath.Join(SabakanDir, "etcd.crt")
	SabakanKeyFile            = filepath.Join(SabakanDir, "etcd.key")
	SabakanConfFile           = filepath.Join(SabakanDir, "config.yml")
	SabactlBin                = "/usr/local/bin/sabactl"
	SabactlBashCompletionFile = filepath.Join(BashCompletionDir, "sabactl")
	SabakanDHCPJSONFile       = filepath.Join(NecoDataDir, "dhcp.json")

	SerfConfFile = filepath.Join(SerfDir, "serf.json")

	CKECertFile          = filepath.Join(CKEDir, "etcd.crt")
	CKEKeyFile           = filepath.Join(CKEDir, "etcd.key")
	CKEConfFile          = filepath.Join(CKEDir, "config.yml")
	CKECLIBin            = "/usr/bin/ckecli"
	CKETemplateFile      = filepath.Join(NecoDataDir, "cke-template.yml")
	CKETemplateFilePre   = filepath.Join(NecoDataDir, "cke-template-pre.yml")
	CKEUserResourceFiles = []string{
		filepath.Join(NecoDataDir, "namespaces.yml"),
		filepath.Join(NecoDataDir, "cilium.yaml"),
		filepath.Join(NecoDataDir, "coil.yaml"),
		filepath.Join(NecoDataDir, "unbound.yml"),
		filepath.Join(NecoDataDir, "squid.yml"),
	CKEUserResourceFilesPre = []string{
		filepath.Join(NecoDataDir, "namespaces.yml"),
		filepath.Join(NecoDataDir, "cilium-pre.yaml"),
		filepath.Join(NecoDataDir, "coil.yaml"),
		filepath.Join(NecoDataDir, "unbound.yml"),
		filepath.Join(NecoDataDir, "squid.yml"),

	NecoCertFile = filepath.Join(NecoDir, "etcd.crt")
	NecoKeyFile  = filepath.Join(NecoDir, "etcd.key")
	NecoConfFile = filepath.Join(NecoDir, "config.yml")
	NecoBin      = "/usr/bin/neco"

	PromtailConfFile = filepath.Join(PromtailDir, "promtail.yaml")

	IgnitionDirectory = filepath.Join(NecoDataDir, "ignitions")

File locations

View Source
var BootImages = []string{"etcd", "promtail", "setup-hw", "sabakan", "serf", "vault"}

BootImages is the list of container image names to be run on boot servers.

View Source
var CurrentArtifacts = ArtifactSet{
	Images: []ContainerImage{
		{Name: "coil", Repository: "", Tag: "2.0.14", Private: false},
		{Name: "bird", Repository: "", Tag: "", Private: false},
		{Name: "chrony", Repository: "", Tag: "", Private: false},
		{Name: "etcd", Repository: "", Tag: "", Private: false},
		{Name: "promtail", Repository: "", Tag: "", Private: false},
		{Name: "sabakan", Repository: "", Tag: "2.13.0", Private: false},
		{Name: "serf", Repository: "", Tag: "", Private: false},
		{Name: "setup-hw", Repository: "", Tag: "1.12.0", Private: true},
		{Name: "squid", Repository: "", Tag: "", Private: false},
		{Name: "vault", Repository: "", Tag: "", Private: false},
		{Name: "cilium", Repository: "", Tag: "", Private: false},
		{Name: "cilium-operator-generic", Repository: "", Tag: "", Private: false},
		{Name: "hubble-relay", Repository: "", Tag: "", Private: false},
		{Name: "cilium-certgen", Repository: "", Tag: "", Private: false},
	Debs: []DebianPackage{
		{Name: "etcdpasswd", Owner: "cybozu-go", Repository: "etcdpasswd", Release: "v1.4.0"},
	OSImage: OSImage{Channel: "stable", Version: "3139.2.1"},
View Source
var SabakanImages = []string{"bird", "chrony", "coil", "serf", "setup-hw", "squid", "cilium", "cilium-operator-generic", "hubble-relay", "cilium-certgen"}

SabakanImages is the list of container image names to be uploaded to sabakan.


func BootNode0IP

func BootNode0IP(lrn int) net.IP

BootNode0IP returns IP address of node0 for bootserver

func DisableService

func DisableService(ctx context.Context, name string) error

DisableService disables the service.

func DisableTimer

func DisableTimer(ctx context.Context, name string) error

DisableTimer disables the timer.

func EtcdClient

func EtcdClient() (*clientv3.Client, error)

EtcdClient returns etcd client for Neco tools.

func EtcdEndpoints

func EtcdEndpoints(lrns []int) []string

EtcdEndpoints returns a list of etcd endpoints for LRNs.

func GetDebianVersion

func GetDebianVersion(pkg string) (string, error)

GetDebianVersion returns debian package version.

func IsActiveService

func IsActiveService(ctx context.Context, name string) (bool, error)

IsActiveService returns true if the service is active.

func MyCluster

func MyCluster() (string, error)

MyCluster returns cluster name of own node

func MyLRN

func MyLRN() (int, error)

MyLRN returns logical rack number of own node

func NewGitHubClient

func NewGitHubClient(c *http.Client) *github.Client

NewGitHubClient returns a properly configured *github.Client.

func OSCodename

func OSCodename() (string, error)

OSCodename returns the OS release codename of the host. See man os-release e.g. bionic, focal

func RestartService

func RestartService(ctx context.Context, name string) error

RestartService restarts the service simply.

func RetryWithSleep

func RetryWithSleep(ctx context.Context, max int, d time.Duration, f func(ctx context.Context) error, logger func(err error)) error

RetryWithSleep invoke f until it succeeds or reach to max.

func ServiceFile

func ServiceFile(name string) string

ServiceFile returns the filesystem path of a systemd service.

func SleepContext

func SleepContext(ctx context.Context, d time.Duration) error

SleepContext sleeps for d, Returned err is not nil if ctx is canceled

func StartService

func StartService(ctx context.Context, name string) error

StartService does following: 1. systemctl daemon-reload 2. systemctl enable NAME.service 3. systemctl start NAME.service

func StartTimer

func StartTimer(ctx context.Context, name string) error

StartTimer does following: 1. systemctl daemon-reload 2. systemctl enable NAME.timer 3. systemctl start NAME.timer

func StopService

func StopService(ctx context.Context, name string) error

StopService stops the service.

func StopTimer

func StopTimer(ctx context.Context, name string) error

StopTimer stops the timer.

func TimerFile

func TimerFile(name string) string

TimerFile returns the filesystem path of a systemd timer.

func UpdateCompleted

func UpdateCompleted(version string, lrns []int, statuses map[int]*UpdateStatus) bool

UpdateCompleted returns true if the current update process has completed successfully.

func VaultClient

func VaultClient(lrn int) (*api.Client, error)

VaultClient returns an authorized Vault client.

If "VAULT_TOKEN" environment variable is set, its value is used as the token to access Vault. Otherwise, this will ask the user Vault username and password.

func WaitVaultLeader

func WaitVaultLeader(ctx context.Context, vc *api.Client) error

WaitVaultLeader waits for Vault to elect a new leader after restart.

Vault wrongly recognizes that the old leader is still a leader after restarting all Vault servers at once. This is probably because the leader information is stored in etcd and Vault references that data to determine the current leader.

While a leader is not yet elected, still Vault servers forward requests to the old non-leader. What's bad is that although the old leader denies the forwarded requests, Vault's Go client library cannot return error.

Specifically, without this workaround, api.Client.Logical.Write() to issue certificates would return (nil, nil)!

func WriteFile

func WriteFile(filename string, data string) error

WriteFile write data to a file.


type ArtifactSet

type ArtifactSet struct {
	// Container image list
	Images []ContainerImage

	// Debian package list
	Debs []DebianPackage

	// OSImage image version
	OSImage OSImage

ArtifactSet represents a set of artifacts.

func (ArtifactSet) FindContainerImage

func (a ArtifactSet) FindContainerImage(name string) (ContainerImage, error)

FindContainerImage finds a ContainerImage from name

func (ArtifactSet) FindDebianPackage

func (a ArtifactSet) FindDebianPackage(name string) (DebianPackage, error)

FindDebianPackage finds a DebianPackage from name

type Bind

type Bind struct {
	Name     string
	Source   string
	Dest     string
	ReadOnly bool

Bind represents a host bind mount rule.

type ContainerImage

type ContainerImage struct {
	// Name is a unique name of this object.
	Name string

	// Repository is a docker repository name.
	Repository string

	// Tag is the image tag.
	Tag string

	// Private indicates that there is a private version of this image.
	Private bool

ContainerImage represents a Docker container image.

func ParseContainerImageName

func ParseContainerImageName(name string) (ContainerImage, error)

ParseContainerImageName parses image name like ""

func (ContainerImage) FullName

func (c ContainerImage) FullName(hasSecret bool) string

FullName returns full container image name. hasSecret should be true if the system has credentials to access private images.

func (ContainerImage) MajorVersion

func (c ContainerImage) MajorVersion() int

MajorVersion returns major version of this image.

func (ContainerImage) MarshalGo

func (c ContainerImage) MarshalGo() string

MarshalGo formats the struct in Go syntax.

func (ContainerImage) NeedAuth

func (c ContainerImage) NeedAuth() bool

NeedAuth returns true if fetching this image needs authentication

type ContainerRuntime

type ContainerRuntime interface {

	// ImageFullName returns the fully-qualified container image name.
	// The result for private images may vary depending on whether the container runtime
	// can access private image repositories.
	ImageFullName(img ContainerImage) string

	// Pull pulls the image.
	Pull(ctx context.Context, img ContainerImage) error

	// Run runs a container for the given image in front.
	Run(ctx context.Context, img ContainerImage, binds []Bind, args []string) error

	// Exec executes the given command in a running container named `name`.
	// The returned error is the error returned by exec.Cmd.Run().
	// If `stdio` is true, the command uses os.Stdin,out,err for I/O.
	Exec(ctx context.Context, name string, stdio bool, command []string) error

	// IsRunning returns true if there is a running container for the image.
	IsRunning(img ContainerImage) (bool, error)

ContainerRuntime defines a set of operations to run containers on boot servers.

func GetContainerRuntime

func GetContainerRuntime(proxy string) (ContainerRuntime, error)

GetContainerRuntime() returns the container runtime for the running server. proxy may be used for some container runtimes.

type ContentsUpdateStatus

type ContentsUpdateStatus struct {
	Version string `json:"version"`
	Success bool   `json:"success"`

ContentsUpdateStatus represents update status of uploaded assets.

type DebianPackage

type DebianPackage struct {
	// Package name.
	Name string

	// Github Owner
	Owner string

	// GitHub repository.
	Repository string

	// GitHub releases (tag name).
	Release string

DebianPackage represents a Debian package hosted in GitHub releases.

func (DebianPackage) MarshalGo

func (deb DebianPackage) MarshalGo() string

MarshalGo formats the struct in Go syntax.

type HardwareType

type HardwareType int

HardwareType represents

const (
	HWTypeNil HardwareType = iota

hardware type

func DetectHardware

func DetectHardware() (HardwareType, error)

DetectHardware detects hardware type.

type ImageFetcher

type ImageFetcher struct {
	// contains filtered or unexported fields

ImageFetcher retrieves Docker image from registries.

func NewImageFetcher

func NewImageFetcher(transport http.RoundTripper, auth authn.Authenticator) ImageFetcher

NewImageFetcher creates a new ImageFetcher. `transport` must not be nil. `auth` can be nil for public repositories.

func (ImageFetcher) GetTarball

func (f ImageFetcher) GetTarball(ctx context.Context, img ContainerImage, w io.Writer) error

GetTarball fetches an image from the registry and write it as a tarball. The tarball can be loaded into Docker with `docker load`.

type OSImage

type OSImage struct {
	Channel string
	Version string

OSImage represents Flatcar Container Linux kernel and initrd images.

func (OSImage) MarshalGo

func (c OSImage) MarshalGo() string

MarshalGo formats the struct in Go syntax.

func (OSImage) URLs

func (c OSImage) URLs() (string, string)

URLs returns kernel and initrd URLs.

type UpdateCondition

type UpdateCondition int

UpdateCondition is the condition of the update process.

func (UpdateCondition) String

func (c UpdateCondition) String() string

String implements io.Stringer

type UpdateRequest

type UpdateRequest struct {
	Version   string    `json:"version"`
	Servers   []int     `json:"servers"`
	Stop      bool      `json:"stop"`
	StartedAt time.Time `json:"started_at"`

UpdateRequest represents request from neco-updater

func (UpdateRequest) IsMember

func (r UpdateRequest) IsMember(lrn int) bool

IsMember returns true if a boot server is the member of this update request.

type UpdateStatus

type UpdateStatus struct {
	Version string          `json:"version"`
	Step    int             `json:"step"`
	Cond    UpdateCondition `json:"cond"`
	Message string          `json:"message"`

UpdateStatus represents status report from neco-worker

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL