confsyncer

README

Conf-Syncer: sync your config files with etcd

Conf-Syncer is a little Go Program that helps keeping config files in sync with a Key Value store. Only etcd v3 is supported but adding other key-value store is just about implementing a simple interface.

Motivations

My TLS certs are fetched dynamically from Let's Encrypt and upgraded every three months. I need to keep these certs in sync between my load-balancer instances and using a distributed file storage was overkill (I'm running Kubernetes clusters on top of AWS and therefore don't have access to ReadManyWriteMany volumes... without deploying a Ceph cluster at least...).

This self-contained program can run on bare-metal or in VMs (using Docker or not) and plays pretty well as an init-container/sidecar container for my Ingress controllers in Kubernetes. The init-container simply pulls the config from the key-value store and dies, the sidecar container keeps the config in sync. If you don't need to be sure that the config files are there when the target container starts for the first time, the sidecar container alone should be just fine).

In the longer run, I'm trying to wrap this in a Kubernetes ThirdPartyResource (a sort of dynamic config map).

Building, Packaging and Distributing

You'll need docker and make in order to build the project, nothing else (we're using a build container), just run make.

You can also build a container with make docker and ship it to a registry of your choice with

DOCKER_IMAGE=my.registry.io/confsyncer DOCKER_TAG=v2 make docker-push

Finally, if you have docker-compose installed, you can get a dev. environment with one etcd container reachable on localhost:20379 and a syncer container. The command to run is make devenv.

Contributing

Contributions are always welcome :)

A containerized development environment based on docker-compose is available with make devenv if you don't want to deal with installing dependencies. It runs an etcd instance that listens on http://localhost:20379.

Every time you bring a change to your code, rerunning make devenv will rebuild what's needed and update the dev. env. (thanks to the old magic of GNU make).

If you want to tail your syncer container's log after the rebuild, simply run make devenv-probe.

All docker-compose commands will also work ;-)

Would you update dependencies in the glide.yaml, run make vendor-update to update them. You can handle dependencies manually with glide.

KVStore fixtures

You can get some data in the etcd container by running make populate-etcd. It should immediately synchronize the config with your disks.

Maintainers

• Étienne Lafarge <etienne.lafarge at gmail.com>

Documentation

Index

• type ConfEvent
• type ConfEventType
• type ConfItem
• type ConfStore
• type DiskStore
  • func NewDiskStore(location string) (cs *DiskStore, err error)
  • func (cs *DiskStore) Close() error
  • func (cs *DiskStore) Delete(key string) error
  • func (cs *DiskStore) List() (data []ConfItem, err error)
  • func (cs *DiskStore) Put(key string, content []byte) error
  • func (cs *DiskStore) Updates() <-chan ConfEvent
• type Etcd3Store
  • func NewEtcd3Store(endpoint, prefix string) (cs *Etcd3Store, err error)
  • func (cs *Etcd3Store) Close() error
  • func (cs *Etcd3Store) Delete(key string) error
  • func (cs *Etcd3Store) List() (data []ConfItem, err error)
  • func (cs *Etcd3Store) Put(key string, value []byte) error
  • func (cs *Etcd3Store) Updates() <-chan ConfEvent
• type Syncer
  • func (s *Syncer) AddLocalConfToRemoteStore() error
  • func (s *Syncer) KeepInSync()
  • func (s *Syncer) PullFromRemote() error

Types

type ConfEvent

type ConfEvent struct {
	Key     string
	Content []byte
	Type    ConfEventType
}

ConfEvent describes a conf change event

type ConfEventType

type ConfEventType int

ConfEventType discriminates config update event types (PUT and DELETE)

const (
	ConfPut ConfEventType = iota
	ConfDelete
)

Conf update event types

type ConfItem

type ConfItem struct {
	Key     string
	Content []byte
}

ConfItem abstracts a config store item

type ConfStore

type ConfStore interface {
	List() (data []ConfItem, err error)
	Close() error
	Updates() <-chan ConfEvent
	Put(key string, value []byte) error
	Delete(key string) error
	SkipNext(key string)
}

ConfStore abstracts a config store (etcd, local disk...)

type DiskStore

type DiskStore struct {
	ConfStore

	Location string
	// contains filtered or unexported fields
}

DiskStore implements a disk backed config store

func NewDiskStore

func NewDiskStore(location string) (cs *DiskStore, err error)

NewDiskStore instantiates a new disk ConfigStore

func (*DiskStore) Close

func (cs *DiskStore) Close() error

Close closes the fsnotify watcher

func (*DiskStore) Delete

func (cs *DiskStore) Delete(key string) error

Delete deletes a given file

func (*DiskStore) List

func (cs *DiskStore) List() (data []ConfItem, err error)

List all conf from local disk

func (*DiskStore) Put

func (cs *DiskStore) Put(key string, content []byte) error

Put writes the content of a given config file

func (*DiskStore) Updates

func (cs *DiskStore) Updates() <-chan ConfEvent

Updates returns the ConfStore update channel

type Etcd3Store

type Etcd3Store struct {
	ConfStore

	Prefix string
	// contains filtered or unexported fields
}

Etcd3Store implements an etcd3-backed config store

func NewEtcd3Store

func NewEtcd3Store(endpoint, prefix string) (cs *Etcd3Store, err error)

NewEtcd3Store creates a new etcd3 config store

func (*Etcd3Store) Close

func (cs *Etcd3Store) Close() error

Close closes the etcd3 client

func (*Etcd3Store) Delete

func (cs *Etcd3Store) Delete(key string) error

Delete deletes a given key

func (*Etcd3Store) List

func (cs *Etcd3Store) List() (data []ConfItem, err error)

List all conf from Etcd3

func (*Etcd3Store) Put

func (cs *Etcd3Store) Put(key string, value []byte) error

Put puts a given value under a given key

func (*Etcd3Store) Updates

func (cs *Etcd3Store) Updates() <-chan ConfEvent

Updates returns the ConfStore update channel

type Syncer

type Syncer struct {
	LocalStore  ConfStore
	RemoteStore ConfStore
	// contains filtered or unexported fields
}

Syncer keeps a local config store in sync with a remote config stores

func (*Syncer) AddLocalConfToRemoteStore

func (s *Syncer) AddLocalConfToRemoteStore() error

AddLocalConfToRemoteStore puts on the remote all the local config files that aren't already in there

func (*Syncer) KeepInSync

func (s *Syncer) KeepInSync()

KeepInSync makes sure all config stores are kept in sync

func (*Syncer) PullFromRemote

func (s *Syncer) PullFromRemote() error

PullFromRemote pulls the config from the remote store without triggering any event. It's useful to initialize the config from the remote single source of truth.