redkeep

README

Red Keep

A MongoDB redundancy keeper.

This project automatically tracks changes and adds redundant fields on references.

Source: http://awoiaf.westeros.org/index.php/File:Red_Keep.jpg

Functionality

Redkeep runs in the background and denormalizes your references for you. Just define what you want to get denormalized, it's easy.

Installation and Usage

go get -d -u github.com/manyminds/redkeep/redkeepcli

Will install the redkeepcli client. Have a look at the example configuration to see how to configure redkeep the way you want. Let's have a look at the configuration of one watch in detail:

    {
      "trackCollection": "application.user",
      "trackFields": ["name", "username"],
      "targetCollection": "application.answer",
      "targetNormalizedField": "meta",
      "triggerReference": "user",
      "behaviourSettings": {
        "cascadeDelete": false
      }
    }

This will watch for changes in the database application and the collection user. If a new answer will be inserted with a reference to application.user the fields name and username will automatically be stored in the newly created answer as the fields meta.name and meta.username.

Documentation

Index

• func BuildInsertQuery(w Watch, command map[string]interface{}) bson.M
• func BuildUpdateQuery(w Watch, command map[string]interface{}) bson.M
• func GetValue(from string, ds interface{}) interface{}
• type BehaviourSettings
• type Configuration
  • func NewConfiguration(configData []byte) (*Configuration, error)
• type InsertTracker
• type Mongo
• type Query
  • func NewOplogQuery(dataset map[string]interface{}) (Query, error)
• type RemoveTracker
• type TailAgent
  • func NewTailAgent(c Configuration) (*TailAgent, error)
  • func NewTailAgentWithStartDate(c Configuration, startTime time.Time) (*TailAgent, error)
  • func (t TailAgent) Tail(quit chan bool, forceRescan bool) error
• type Tracker
  • func NewChangeTracker(session *mgo.Session) Tracker
• type UpdateTracker
• type Watch

Functions

func BuildInsertQuery

func BuildInsertQuery(w Watch, command map[string]interface{}) bson.M

BuildInsertQuery generates the query

func BuildUpdateQuery

func BuildUpdateQuery(w Watch, command map[string]interface{}) bson.M

BuildUpdateQuery generates the query

func GetValue

func GetValue(from string, ds interface{}) interface{}

GetValue works like this: from must be a selector like user.comment.author GetValue then looks recursively for that element therefore all of the following return values are possible map[string]interface{} nil string or basic mongodb types

Types

type BehaviourSettings

type BehaviourSettings struct {
	CascadeDelete bool `json:"cascadeDelete"`
}

BehaviourSettings can define how one specific watch handles special cases

type Configuration

type Configuration struct {
	Mongo   Mongo   `json:"mongo" validate:"required"`
	Watches []Watch `json:"watches" validate:"required,gt=0,dive"`
}

Configuration for red keep

func NewConfiguration

func NewConfiguration(configData []byte) (*Configuration, error)

NewConfiguration loads a configuration from data if it is not valid json, it will return an error

type InsertTracker

type InsertTracker interface {
	HandleInsert(
		w Watch,
		command map[string]interface{},
		originRef mgo.DBRef,
	)
}

InsertTracker can handle inserts

type Mongo

type Mongo struct {
	ConnectionURI string `json:"connectionURI" validate:"required,gt=0"`
}

Mongo is a config struct that changes the way the client connects. ConnectionURI should be a string identifying your cluster if you have to slaves and one master it would be like slave-01:27018,slave-02:27018,master:27018 where slave-01 is either a hostname or an ip

type Query

type Query interface {
	DB() string
	C() string
	OP() string
}

Query represents a mongodb oplog query

func NewOplogQuery

func NewOplogQuery(dataset map[string]interface{}) (Query, error)

NewOplogQuery generates a new query object from the given dataset

type RemoveTracker

type RemoveTracker interface {
	HandleRemove(
		w Watch,
		command map[string]interface{},
		selector map[string]interface{},
	)
}

RemoveTracker can handle removes

type TailAgent

type TailAgent struct {
	// contains filtered or unexported fields
}

TailAgent the worker that tails the database

func NewTailAgent

func NewTailAgent(c Configuration) (*TailAgent, error)

NewTailAgent will generate a new tail agent

func NewTailAgentWithStartDate

func NewTailAgentWithStartDate(c Configuration, startTime time.Time) (*TailAgent, error)

NewTailAgentWithStartDate will start

func (TailAgent) Tail

func (t TailAgent) Tail(quit chan bool, forceRescan bool) error

Tail will start an inifite look that tails the oplog as long as the channel does not get any input forceRescan (Default false) will update anything from the lowest oplog timestamp again. Can cause many redundant writes depending on your oplog size.

type Tracker

type Tracker interface {
	RemoveTracker
	UpdateTracker
	InsertTracker
}

Tracker handles changes in the oplog it is a combination of CRUD Tracker Remove/Update/Create/Delete

func NewChangeTracker

func NewChangeTracker(session *mgo.Session) Tracker

NewChangeTracker is the default tracker implementation of redkeep

type UpdateTracker

type UpdateTracker interface {
	HandleUpdate(
		w Watch,
		command map[string]interface{},
		selector map[string]interface{},
	)
}

UpdateTracker can handle updates

type Watch

type Watch struct {
	//TODO validate collections to be in this scheme: database.collection
	TrackCollection       string            `json:"trackCollection" validate:"required,gt=0"`
	TrackFields           []string          `json:"trackFields" validate:"required,min=1,dive,min=1"`
	TargetCollection      string            `json:"targetCollection" validate:"required,min=1"`
	TargetNormalizedField string            `json:"targetNormalizedField" validate:"required,min=1"`
	TriggerReference      string            `json:"triggerReference" validate:"required,min=1"`
	BehaviourSettings     BehaviourSettings `json:"behaviourSettings"`
}

Watch defines one watch that redkeep will do for you