grim

README

#Grim: Dead simple build server

Grim is the "GitHub Responder In MediaMath". We liked the acronym and awkwardly filled in the details to fit it. In short, it is a task runner that is triggered by GitHub push/pull request hooks that is intended as a much simpler and easy-to-use build server than the more modular alternatives (eg. Jenkins).

On start up, Grim will:

1. Create (or reuse) an Amazon SQS queue with the name specified in its config file as GrimQueueName
2. Detect which GitHub repositories it is configured to work with
3. Create (or reuse) an Amazon SNS topic for each repository
4. Configure each created topic to push to the Grim queue
5. Configure each repositories' AmazonSNS service to push hook updates (push, pull_request) to the topic

An example including two repositories watched by three Grimd's (one in EC2 and two MacBookPros).

Each GitHub repo can push to exactly one SNS topic. Multiple SQS queues can subscribe to one topic and multiple Grim instances can read from the same SQS queue. If a Grim instance isn't configured to respond to the repo specified in the hook it silently ignores the event.

##Installation

1. Get grimd

wget https://artifactory.mediamath.com/artifactory/libs-release-global/com/mediamath/grim/grimd/[RELEASE]/grimd-[RELEASE].zip

2. Global Configuration

Grim tries to honor the conventional Unix filesystem hierarchy as much as possible. Configuration files are by default found in /etc/grim. You may override that default by specifying --config-root [some dir], more briefly -c [some dir] or by setting the GRIM_CONFIG_ROOT environment variable. Inside that directory there is expected to be a config.json that specifies the other paths used as well as global defaults. Here is an example:

{
	"GrimQueueName": "grim-queue",
	"ResultRoot": "/var/log/grim",
	"WorkspaceRoot": "/var/tmp/grim",
	"AWSRegion": "us-east-1",
	"AWSKey": "xxxx",
	"AWSSecret": "xxxx",
	"GitHubToken": "xxxx",
	"HipChatToken": "xxxx"
}

If you don't configure GrimQueueName, ResultRoot or WorkspaceRoot Grim will use default values. The AWS credentials supplied must be able to create and modify SNS topics and SQS queues.

Required GitHub token scopes

• write:repo_hook to be able to create/edit repository hooks
• repo:status to be able set commit statuses
• repo to be able to download the repo

3. Repository Configuration

In order for Grim to respond to GitHub events it needs subdirectories to be made in the configuration root. Inside those subdirectories should be a config.json and optionally a build.sh. Here is an example directory structure:

/etc/grim
/etc/grim/config.json
/etc/grim/MediaMath
/etc/grim/MediaMath/grim
/etc/grim/MediaMath/grim/config.json
/etc/grim/MediaMath/grim/build.sh

The file config.json can have an empty JSON object or have the following fields:

{
	"GitHubToken": "xxxx",
	"HipChatToken": "xxxx"
	"HipChatRoom": "xxxx",
	"PathToCloneIn": "/go/src/github.com/MediaMath/grim"
}

The GitHub and HipChat tokens will override the global ones if present. The HipChat room is optional and if present will indicate that status messages will go to that room. The field PathToCloneIn is relative to the workspace that was created for this build.

Build script location

Grim will look for a build script first in the configuration directory for the repo as build.sh and failing that in the root of the cloned repo as either .grim_build.sh or grim_build.sh.

The environment variables available to this script are documented here.

Environment Variables

CLONE_PATH= the path relative to the workspace the repo is cloned in 
GH_EVENT_NAME= either 'push', 'pull_request' or '' (for manual builds)
GH_ACTION= the sub action of a pull request (eg. 'opened', 'closed', or 'reopened', 'synchronize') or blank for other event types
GH_USER_NAME= the user initiating the event
GH_OWNER= the owner part of a repo (eg. 'MediaMath')
GH_REPO= the name of a repo (eg. 'grim')
GH_TARGET= the branch that a commit was merged to
GH_REF= the ref to build
GH_STATUS_REF= the ref to set the status of
GH_URL= the GitHub URL to find the changes at

Documentation

Overview

Package grim is the "GitHub Responder In MediaMath". We liked the acronym and awkwardly filled in the details to fit it. In short, it is a task runner that is triggered by GitHub push/pull request hooks that is intended as a much simpler and easy-to-use build server than the more modular alternatives (eg. Jenkins). grim provides the library functions to support this use case. grimd is a daemon process that uses the grim library.

Index

• Constants
• Variables
• func IsFatal(err error) bool
• type BuildTimeout
• type Error
• type Instance
  • func (i *Instance) BuildNextInGrimQueue(logger *log.Logger) error
  • func (i *Instance) BuildRef(owner, repo, ref string, logger *log.Logger) error
  • func (i *Instance) PrepareGrimQueue(logger *log.Logger) error
  • func (i *Instance) PrepareRepos() error
  • func (i *Instance) SetConfigRoot(path string)

Constants

const (
	RSPending refStatusState = "pending"
	RSSuccess refStatusState = "success"
	RSError   refStatusState = "error"
	RSFailure refStatusState = "failure"
)

These statuses model the statuses mentioned here: https://developer.github.com/v3/repos/statuses/#create-a-status

const (
	ColorYellow messageColor = "yellow"
	ColorRed    messageColor = "red"
	ColorGreen  messageColor = "green"
	ColorPurple messageColor = "purple"
	ColorGray   messageColor = "gray"
	ColorRandom messageColor = "random"
)

These colors model the colors mentioned here: https://www.hipchat.com/docs/api/method/rooms/message

Variables

var GrimError = &standardGrimNotification{RSError, ColorGray, func(c *effectiveConfig) string { return c.errorTemplate }}

GrimError is the notification used for builds that cannot be run correctly.

var GrimFailure = &standardGrimNotification{RSFailure, ColorRed, func(c *effectiveConfig) string { return c.failureTemplate }}

GrimFailure is the notification used when builds fail.

var GrimPending = &standardGrimNotification{RSPending, ColorYellow, func(c *effectiveConfig) string { return c.pendingTemplate }}

GrimPending is the notification used for pending builds.

var GrimSuccess = &standardGrimNotification{RSSuccess, ColorGreen, func(c *effectiveConfig) string { return c.successTemplate }}

GrimSuccess is the notification used when builds succeed.

Functions

func IsFatal

func IsFatal(err error) bool

IsFatal will determine if a Grim error is recoverable.

Types

type BuildTimeout

type BuildTimeout interface {
	BuildTimeout() time.Duration
}

BuildTimeout is a "safe" way to get the timeout configured for builds. It will ensure non-zero build timeouts.

type Error

type Error interface {
	IsFatal() bool
}

Error models failures in Grim methods

type Instance

type Instance struct {
	// contains filtered or unexported fields
}

Instance models the state of a configured Grim instance.

func (*Instance) BuildNextInGrimQueue

func (i *Instance) BuildNextInGrimQueue(logger *log.Logger) error

BuildNextInGrimQueue creates or reuses an SQS queue as a source of work.

func (*Instance) BuildRef

func (i *Instance) BuildRef(owner, repo, ref string, logger *log.Logger) error

BuildRef builds a git ref immediately.

func (*Instance) PrepareGrimQueue

func (i *Instance) PrepareGrimQueue(logger *log.Logger) error

PrepareGrimQueue creates or reuses the Amazon SQS queue named in the config.

func (*Instance) PrepareRepos

func (i *Instance) PrepareRepos() error

PrepareRepos discovers all repos that are configured then sets up SNS and GitHub. It is an error to call this without calling PrepareGrimQueue first.

func (*Instance) SetConfigRoot

func (i *Instance) SetConfigRoot(path string)

SetConfigRoot sets the base path of the configuration directory and clears any previously read config values from memory.