cedar

README

=================================
``cedar`` -- Build Data Warehouse
=================================

Overview
--------

Cedar is a collection of data-focused tools for data generated by and used in Evergreen builds. It is not a part of
Evergreen, but serves as a companion service and tool kit. In particular, Cedar focuses on the "offline" data processing
cases, and providing access to data to support external user interfaces (e.g. cli tools, data visualizations, etc.)

Build operations generate data, and Evergreen stores most of this data directly (e.g. the outcome of tasks, artifacts
produced by builds). There's other data, specific to particular projects, that is less naturally structured or less
specifically linked to specific execution events. Additionally, Cedar provides a platform to collect and analyze data
*about* an Evergreen deployment in the large.

Components
~~~~~~~~~~

- A simple JSON-only REST web service for data access (see the ``rest`` package).

- A gRPC service for data storage (see the ``rpc`` package).

- A command line tool that provides a REST client for common operations, as well as independent data collection and
  exploration tools (see the ``operations`` package).

- An offline task processing framework based on `amboy <https://github.com/mongodb/amboy/>`_ for application-level data
  aggregation operations (see the ``units`` package).

Data Types ~~~~~~~~~~

- Logs for Evergreen tasks.

- Test result logs and metadata for Evergreen tasks.

- Host system metrics for tasks.

- A simple log storage system, as a proof of concept. Logs stream from the client to the cedar application which stores
  metadata about the logs and then saves data to S3. Out of band, jobs can process log data.

Use ---

#. Download and compile cedar. I prefer something like: ::

     go get github.com/evergreen-ci/cedar cd $GOPATH/src/github.com/evergreen-ci/cedar
     make build
     pushd /usr/local/bin
     ln -s $GOPATH/src/github.com/evergreen-ci/cedar/build/cedar
     popd

#. Explore the help menus in the ``cedar`` binary: ::

     cedar --help
     cedar service --help
     cedar client --help
     cedar worker --help

#. Enjoy!

Development -----------

The cedar project uses a ``makefile`` to coordinate testing. Use the following command to build the cedar binary: ::

  make cedar

The artifact is at ``build/cedar``. The makefile provides the following targets:

``test`` Runs all tests, sequentially, for all packages.

``test-<package>`` Runs all tests for a specific package

``lint``, ``lint-<package>`` Installs and runs the ``gometaliter`` with appropriate settings to lint the project.

Documentation

Overview

Package cedar holds a a number of application level constants and shared resources for the cedar application.

Index

• Constants
• Variables
• func GetUserMiddlewareConfiguration() gimlet.UserMiddlewareConfiguration
• func SetEnvironment(env Environment)
• type CloserFunc
• type Configuration
  • func GetSessionWithConfig(env Environment) (*Configuration, db.Session, error)
  • func (c *Configuration) GetQueueGroupOptions() queue.MongoDBOptions
  • func (c *Configuration) GetQueueOptions() queue.MongoDBOptions
  • func (c *Configuration) HasAuth() bool
  • func (c *Configuration) Validate() error
• type Environment
  • func GetEnvironment() Environment
  • func NewEnvironment(ctx context.Context, name string, conf *Configuration) (Environment, error)
• type EnvironmentCache
• type Stat

Constants

const (
	// Auth constants.
	AuthTokenCookie        = "cedar-token"
	APIUserHeader          = "Api-User"
	APIKeyHeader           = "Api-Key"
	EvergreenAPIUserHeader = "Evergreen-Api-User"
	EvergreenAPIKeyHeader  = "Evergreen-Api-Key"
	TokenExpireAfter       = time.Hour

	// Version requester types.
	PatchVersionRequester       = "patch_request"
	GithubPRRequester           = "github_pull_request"
	GitTagRequester             = "git_tag_request"
	RepotrackerVersionRequester = "gitter_request"
	TriggerRequester            = "trigger_request"
	MergeTestRequester          = "merge_test"
	AdHocRequester              = "ad_hoc"

	// Stats cache names.
	StatsCacheBuildlogger = "buildlogger"
	StatsCacheTestResults = "test_results"
	StatsCachePerf        = "perf"
)

const (
	ShortDateFormat = "2006-01-02T15:04"
)

Variables

var (
	// Convenience slice for patch requester types.
	PatchRequesters = []string{
		PatchVersionRequester,
		GithubPRRequester,
		MergeTestRequester,
	}

	// Convenience slice for slice cache names.
	StatsCacheNames = []string{
		StatsCacheBuildlogger,
		StatsCacheTestResults,
		StatsCachePerf,
	}
)

var BuildRevision = ""

BuildRevision stores the commit in the git repository at build time and is specified with -ldflags at build time.

Functions

func GetUserMiddlewareConfiguration

func GetUserMiddlewareConfiguration() gimlet.UserMiddlewareConfiguration

GetUserMiddlewareConfiguration returns the gimlet user middleware config with cedar's global auth constants.

func SetEnvironment

func SetEnvironment(env Environment)

Types

type CloserFunc

type CloserFunc func(context.Context) error

type Configuration

type Configuration struct {
	BucketName              string
	DatabaseName            string
	QueueName               string
	MongoDBURI              string
	MongoDBDialTimeout      time.Duration
	SocketTimeout           time.Duration
	DisableLocalQueue       bool
	DisableRemoteQueue      bool
	DisableRemoteQueueGroup bool
	DisableCache            bool
	NumWorkers              int
	DBUser                  string
	DBPwd                   string
}

Configuration defines configuration settings to initialize the global environment without the presence of a database to store the settings.

func GetSessionWithConfig

func GetSessionWithConfig(env Environment) (*Configuration, db.Session, error)

func (*Configuration) GetQueueGroupOptions

func (c *Configuration) GetQueueGroupOptions() queue.MongoDBOptions

GetQueueGroupOptions returns the options to initialize a MongoDB-backed Amboy queue group.

func (*Configuration) GetQueueOptions

func (c *Configuration) GetQueueOptions() queue.MongoDBOptions

GetQueueOptions returns the options to initialize a MongoDB-backed Amboy queue.

func (*Configuration) HasAuth

func (c *Configuration) HasAuth() bool

HasAuth returns whether or not the database uses authentication.

func (*Configuration) Validate

func (c *Configuration) Validate() error

Validate checks that all the required fields are set and sets defaults for unset optional fields.

type Environment

type Environment interface {
	GetConf() *Configuration
	GetCache() (EnvironmentCache, bool)
	Context() (context.Context, context.CancelFunc)

	// GetQueue retrieves the application's shared queue, which is cached
	// for easy access from within units or inside of requests or command
	// line operations.
	GetRemoteQueue() amboy.Queue
	SetRemoteQueue(amboy.Queue) error
	GetRemoteManager() management.Manager
	GetLocalQueue() amboy.Queue
	SetLocalQueue(amboy.Queue) error
	GetRemoteQueueGroup() amboy.QueueGroup

	GetSession() db.Session
	GetClient() *mongo.Client
	GetDB() *mongo.Database
	Jasper() jasper.Manager

	GetServerCertVersion() int
	SetServerCertVersion(i int)

	// GetStatsCache returns the cache corresponding to the string.
	GetStatsCache(string) *statsCache

	RegisterCloser(string, CloserFunc)
	Close(context.Context) error
}

Environment objects provide access to shared configuration and state, in a way that you can isolate and test for in

func GetEnvironment

func GetEnvironment() Environment

func NewEnvironment

func NewEnvironment(ctx context.Context, name string, conf *Configuration) (Environment, error)

type EnvironmentCache

type EnvironmentCache interface {
	// PutNew adds new (key, value) pair to the cache and returns whether
	// the operation was successful or not.
	PutNew(string, interface{}) bool
	// RegisterUpdater registers a new cache value updater for the given
	// key and returns whether the operation was successful or not. The
	// updater should listen for updates on the given channel and use the
	// context and cancel function for signaling.
	RegisterUpdater(context.Context, context.CancelFunc, string, chan interface{}) bool
	// Get returns the value of the given key and if the key exists in the
	// cache.
	Get(string) (interface{}, bool)
	// Delete removes the given key from the cache.
	Delete(string)
}

EnvironmentCache provides thread-safe, in-memory access to data persisted elsewhere, such as on disk or a database.

type Stat

type Stat struct {
	Count   int
	Project string
	Version string
	TaskID  string
}

Stat represents a count to add to the cache for a particular project/version/taskID combination.