config

package module
v0.0.0-...-4c14a5a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 13, 2020 License: Apache-2.0 Imports: 18 Imported by: 0

README

config

Golang process config synchronization using etcd

Documentation

Overview

Package config implements a configuration server using git and zookeeper.

Besides maintaining structured data describing how systems should work together, a good configuration server needs to have the following characteristics:

1. Version control. Should be able to revert to a previous version shall there be any error after a push. 2. Fast, simple, and reliable notification to thousands or tens of thousands servers. 3. Flexible to describe complex systems. 4. Ease of use

For the above reasons, we chose to implement our config system with two components that are already widely used with a rock solid foundation: git, and zookeeper.

It is easy to see that git can handle require 1, 3, and 4, while zookeeper satisfies 2, and 4. The system works as follows:

1. Configurations are stored in a git repo. You can have one file for the whole systems, or you can have hundreds or thousands of files each detailing a part of the whole system. It doesn't matter. The config system deal with repo, and its commits. The sysadmin maintain the repo using whatever git tool he/she choose. Changes to the git repo are not saved to config system. So just use it as a normal repo, branch, edit, etc as you would for a normal repo.

2. For changes to be pushed to our config system, you need to call csi/service/config/pusher/pusher. pusher works on the HEAD of the repo. Whatever you have checked out, when pusher is called, will be pushed onto config system. pusher will update data in a zookeeper ensemble that acts as the persistent, and notification layer. When pusher is called, it is given a root path, which is the root path for the configuration information. This way, it is ok if you just dedicate a subdirectory of your repo for configuration, instead of dedicating a completely new one. All descendent files of the root directory will be saved onto zookeeper, e.g. a/b/c/d will be set in zookeeper as znode a/b/c/d where the content of znode a/b/c/d is the content of file a/b/c/d. Additionally, pusher will collect repo and commit info, encode them in a json structure and store it in the root znode for this configuration system. The json structure contains repo name, branch name, last commit hash, committer email, timestamp, and a files modified in the last commit.

3. The information stored in step 2 on the root znode is important. It allows a client of the configuration system to just listen to one znode, and gets notified of not only a change has happened, but what file(s) got changed, and decided whether or not it should update. This job is done by csi/service/config/client.go. Client.go allows one to register to receive event on any file change.

4. Pusher requires "git" to be in the PATH, as it calls git related command extensively. The following is a list of git commands that pusher uses to find out various repo related info:

# show repo name 00:57 $ basename `git rev-parse --show-toplevel` csi

# show HEAD branch name 00:56 $ git rev-parse --abbrev-ref HEAD master

# show HEAD commit hash 00:54 $ git rev-parse HEAD 6e390e716805599f8f694a5f654649d92ec638f6

# show files affected in a commit 01:02 $ git diff-tree --no-commit-id --name-status -r 6e390e716805599f8f694a5f654649d92ec638f6 common/types/user/login_record.pb.go external/oauth/cache.go make_go.sh pp/frontend/main.go .....

# get timestamp, committer email and subject of a commit 03:03 $ git show --pretty="format:%at:%ce:%s" 6e390e716805599f8f694a5f654649d92ec638f6 | head -n 1 1394199814:alien_lien@htc.com:Add 1) Account Manager and 2) User Management Service

5. Client can listen on the configuration change with a regex string that specifies which files it is interested. It will be notified when any of files that matches the regex has changed. Note: this is achieved by regex match the changed files in the commit log with is published to the root znode.

Index

Constants

View Source 
const (
	// PkgName is for creating module counter
	PkgName = "config"
)

Variables

This section is empty.

Functions

func Stat 

func Stat(counter stats.Client) func(*clientImpl) error

Stat sets counter client

Types

type Client 

type Client interface {
	// ConfigInfo returns the ConfigInfo protobuf struct that
	// contains infomation w.r.t to repo and last commit
	ConfigInfo() ConfigInfo

	// Get request the content of file at path. If the path doesn't
	// exists then ok returns false
	Get(path string) ([]byte, error)

	// AddListener returns a chan that when any file that matches
	// the pathRegEx changes, the ModifiedFile will be sent over
	// through the channel
	AddListener(pathRegEx *regexp.Regexp) *chan ModifiedFile

	// Watch watches change of a file and invokes callback. It also invokes callback for the
	// first time and return error if there's one.
	Watch(path string, callback func([]byte) error, errChan chan<- error) error

	// RemoveListener remove a listener channel
	RemoveListener(ch *chan ModifiedFile)

	// List lists content under certain path
	List(path string) (map[string][]byte, error)

	// Stop is to stop client
	Stop() error
}

Client provides a interface for configuration service

func NewClient 

func NewClient(conn *etcd.Client, root string, options ...func(*clientImpl) error) (Client, error)

NewClient return a new Client with option funtions. conn: a etcd client, root: the root path that contains the config tree

func NewFileConfig 

func NewFileConfig(root string, uncommitOnly bool) (Client, error)

NewFileConfig return filepath instance for local repo. Optionally you can set uncommitOnly to true which causes List() method only return a subset of uncommitted files.

func NewMemConfig 

func NewMemConfig(files map[string][]byte) Client

NewMemConfig creates a memory mapped config

type CommitInfo 

type CommitInfo struct {
	TimeStamp      int64  `json:"timestamp,omitempty"`
	CommitterEmail string `json:"committerEmail,omitempty"`
	Subject        string `json:"subject,omitempty"`
}

CommitInfo contains git commit info

type ConfigInfo 

type ConfigInfo struct {
	Repo     string         `json:"repo,omitempty"`
	Branch   string         `json:"branch,omitempty"`
	Version  string         `json:"version,omitempty"`
	Commit   CommitInfo     `json:"commit,omitempty"`
	ModFiles []ModifiedFile `json:"modfiles,omitempty"`
	// PathToRepo is the path to git root directory
	PathToRepo string `json:"path_to_repo,omitempty"`
}

ConfigInfo defines a configuration service information

type FileConfig 

type FileConfig struct {
	// contains filtered or unexported fields
}

FileConfig is used to test config file in local

func (FileConfig) AddListener 

func (d FileConfig) AddListener(pathRegEx *regexp.Regexp) *chan ModifiedFile

func (FileConfig) ConfigInfo 

func (d FileConfig) ConfigInfo() ConfigInfo

func (*FileConfig) Get 

func (l *FileConfig) Get(path string) ([]byte, error)

Get get the file from local directory under root

func (*FileConfig) List 

func (l *FileConfig) List(path string) (map[string][]byte, error)

List list all the files of the path

func (FileConfig) RemoveListener 

func (d FileConfig) RemoveListener(ch *chan ModifiedFile)

RemoveListener remove a listener channel

func (*FileConfig) Stop 

func (l *FileConfig) Stop() error

Stop is to stop config client

func (FileConfig) Watch 

func (d FileConfig) Watch(string, func([]byte) error, chan<- error) error

type MemConfig 

type MemConfig struct {
	// contains filtered or unexported fields
}

MemConfig is a memroy mapped config client

func (MemConfig) AddListener 

func (d MemConfig) AddListener(pathRegEx *regexp.Regexp) *chan ModifiedFile

func (MemConfig) ConfigInfo 

func (d MemConfig) ConfigInfo() ConfigInfo

func (*MemConfig) Get 

func (m *MemConfig) Get(filename string) ([]byte, error)

Get gets file content

func (*MemConfig) List 

func (m *MemConfig) List(path string) (map[string][]byte, error)

List lists files under the path TODO: it only returns file without any directories.

func (MemConfig) RemoveListener 

func (d MemConfig) RemoveListener(ch *chan ModifiedFile)

RemoveListener remove a listener channel

func (*MemConfig) Stop 

func (m *MemConfig) Stop() error

Stop is to stop config client

func (MemConfig) Watch 

func (d MemConfig) Watch(string, func([]byte) error, chan<- error) error

type MockClient 

type MockClient struct {
	mock.Mock
}

MockClient is the mock struct

func (*MockClient) AddListener 

func (m *MockClient) AddListener(pathRegEx *regexp.Regexp) *chan ModifiedFile

AddListener is the mock function.

func (*MockClient) ConfigInfo 

func (m *MockClient) ConfigInfo() ConfigInfo

ConfigInfo is the mock function.

func (*MockClient) Get 

func (m *MockClient) Get(path string) ([]byte, error)

Get is the mock function.

func (*MockClient) List 

func (m *MockClient) List(path string) (map[string][]byte, error)

List is the mock function.

func (*MockClient) RemoveListener 

func (m *MockClient) RemoveListener(ch *chan ModifiedFile)

RemoveListener is the mock function.

func (*MockClient) Stop 

func (m *MockClient) Stop() error

Stop is to stop client

func (*MockClient) Watch 

func (m *MockClient) Watch(path string, callback func([]byte) error, errChan chan<- error) error

Watch watches change of a file and invokes callback. It also invokes callback for the

type ModifiedFile 

type ModifiedFile struct {
	Op   string `json:"op,omitempty"`
	Path string `json:"path,omitempty"`
}

ModifiedFile contains a file and its modify operation

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.