vulcanizer

README

vulcanizer

GitHub's ops focused Elasticsearch library

This project is a golang library for interacting with an Elasticsearch cluster. It's goal is to provide a high level API to help with common tasks that are associated with operating an Elasticsearch cluster such as querying health status of the cluster, migrating data off of nodes, updating cluster settings, etc.

This project does not aim to be a fully-featured API client for querying or indexing to Elasticsearch.

Go API

You can perform custom operations in your Go application.

import "github.com/github/vulcanizer"

v = vulcanizer.NewClient("localhost", 9200)
oldSetting, newSetting, err := v.SetSetting("indices.recovery.max_bytes_per_sec", "1000mb")

Command line application

This project produces a vulcanizer binary that is a command line application that can be used to manage your Elasticsearch cluster.

$ vulcanizer -h
Usage:
  vulcanizer [command]

Available Commands:
  allocation  Set shard allocation on the cluster.
  drain       Drain a server or see what servers are draining.
  fill        Fill servers with data, removing shard allocation exclusion rules.
  health      Display the health of the cluster.
  help        Help about any command
  indices     Display the indices of the cluster.
  nodes       Display the nodes of the cluster.
  setting     Interact with cluster settings.
  settings    Display all the settings of the cluster.
  snapshot    Interact with a specific snapshot.
  snapshots   Display the snapshots of the cluster.

Flags:
  -c, --cluster string      Cluster to connect to defined in config file
  -f, --configFile string   Configuration file to read in (default to "~/.vulcanizer.yaml")
  -h, --help                help for vulcanizer
      --host string         Host to connect to (default "localhost")
  -p, --port int            Port to connect to (default 9200)

Use "vulcanizer [command] --help" for more information about a command.

Roadmap and future releases

The proposed future for vulcanizer can be found in our ROADMAP.

Configuration and connection information

All commands take --cluster <name> to look up information in a configuration file in ~/.vulcanizer.yaml. The configuration should be in the form of

local:
  host: localhost
  port: 9200
staging:
  host: 10.10.2.1
  port: 9201
production:
  host: 10.10.1.1
  port: 9202

Alternatively, all commands take --host and --port for the connection information.

For example:

# Query for cluster health on the "local" cluster
vulcanizer health --cluster local

# Query for nodes against the node 10.10.2.1 and port 9202
vulcanizer nodes --host 10.10.2.1 --port 9202

Development

./script/build will compile the project and install the vulcanizer binary to $GOPATH/bin.

./script/test will run the tests in the project.

Supported Elasticsearch versions

To be determined

Name

Vulcanization is the process of making rubber more elastic, so vulcanizer is the library that makes Elasticsearch easier to work with!

Project status

This project is under active development.

Contributing

This repository is open to contributions. Please also see code of conduct

To get up and running, install the project into your $GOPATH and run the set up scripts.

go get github.com/github/vulcanizer

cd $GOPATH/src/github.com/github/vulcanizer

./script/bootstrap
./script/test

And the test suite should execute correctly.

License

This project is released under the MIT LICENSE. Please note it includes 3rd party dependencies release under their own licenses; these are found under vendor. When using the GitHub logos, be sure to follow the GitHub logo guidelines.

Authors

Authored by GitHub Engineering

Documentation

Index

• type Client
  • func NewClient(host string, port int) *Client
  • func (c *Client) DeleteSnapshot(repository string, snapshot string) error
  • func (c *Client) DrainServer(serverToDrain string) (ExcludeSettings, error)
  • func (c *Client) FillAll() (ExcludeSettings, error)
  • func (c *Client) FillOneServer(serverToFill string) (ExcludeSettings, error)
  • func (c *Client) GetClusterExcludeSettings() (ExcludeSettings, error)
  • func (c *Client) GetHealth() ([]ClusterHealth, error)
  • func (c *Client) GetIndices() ([]Index, error)
  • func (c *Client) GetNodes() ([]Node, error)
  • func (c *Client) GetSettings() (ClusterSettings, error)
  • func (c *Client) GetSnapshotStatus(repository string, snapshot string) (Snapshot, error)
  • func (c *Client) GetSnapshots(repository string) ([]Snapshot, error)
  • func (c *Client) SetAllocation(allocation string) (string, error)
  • func (c *Client) SetSetting(setting string, value string) (string, string, error)
  • func (c *Client) VerifyRepository(repository string) (bool, error)
• type ClusterHealth
• type ClusterSetting
• type ClusterSettings
• type ExcludeSettings
• type Index
• type Node
• type Snapshot

Types

type Client

type Client struct {
	Host string
	Port int
}

Hold connection information to a Elasticsearch cluster.

func NewClient

func NewClient(host string, port int) *Client

Initialize a new vulcanizer client to use.

func (*Client) DeleteSnapshot

func (c *Client) DeleteSnapshot(repository string, snapshot string) error

Delete a snapshot

Use case: You want to delete older snapshots so that they don't take up extra space.

func (*Client) DrainServer

func (c *Client) DrainServer(serverToDrain string) (ExcludeSettings, error)

Set shard allocation exclusion rules such that the Elasticsearch node with the name `serverToDrain` is excluded. This should cause Elasticsearch to migrate shards away from that node.

Use case: You need to restart an Elasticsearch node. In order to do so safely, you should migrate data away from it. Calling `DrainServer` with the node name will move data off of the specified node.

func (*Client) FillAll

func (c *Client) FillAll() (ExcludeSettings, error)

Removes all shard allocation exclusion rules.

Use case: You had been performing maintenance on a number of Elasticsearch nodes. They are all ready to receive data again. Calling `FillAll` will remove all the allocation exclusion rules on the cluster, allowing Elasticsearch to freely allocate shards on the previously excluded nodes.

func (*Client) FillOneServer

func (c *Client) FillOneServer(serverToFill string) (ExcludeSettings, error)

Set shard allocation exclusion rules such that the Elasticsearch node with the name `serverToFill` is no longer being excluded. This should cause Elasticsearch to migrate shards to that node.

Use case: You have completed maintenance on an Elasticsearch node and it's ready to hold data again. Calling `FillOneServer` with the node name will remove that node name from the shard exclusion rules and allow data to be relocated onto the node.

func (*Client) GetClusterExcludeSettings

func (c *Client) GetClusterExcludeSettings() (ExcludeSettings, error)

Get current cluster settings for shard allocation exclusion rules.

func (*Client) GetHealth

func (c *Client) GetHealth() ([]ClusterHealth, error)

Get the health of the cluster.

Use case: You want to see information needed to determine if the Elasticsearch cluster is healthy (green) or not (yellow/red).

func (*Client) GetIndices

func (c *Client) GetIndices() ([]Index, error)

Get all the indices in the cluster.

Use case: You want to see some basic info on all the indices of the cluster.

func (*Client) GetNodes

func (c *Client) GetNodes() ([]Node, error)

Get all the nodes in the cluster.

Use case: You want to see what nodes Elasticsearch considers part of the cluster.

func (*Client) GetSettings

func (c *Client) GetSettings() (ClusterSettings, error)

Get all the persistent and transient cluster settings.

Use case: You want to see the current settings in the cluster.

func (*Client) GetSnapshotStatus

func (c *Client) GetSnapshotStatus(repository string, snapshot string) (Snapshot, error)

Get detailed information about a particular snapshot.

Use case: You had a snapshot fail and you want to see the reason why and what shards/nodes the error occurred on.

func (*Client) GetSnapshots

func (c *Client) GetSnapshots(repository string) ([]Snapshot, error)

List the snapshots of the given repository.

Use case: You want to see information on snapshots in a repository.

func (*Client) SetAllocation

func (c *Client) SetAllocation(allocation string) (string, error)

Enables or disables allocation for the cluster.

Use case: You are performing an operation the cluster where nodes may be dropping in and out. Elasticsearch will typically try to rebalance immediately but you want the cluster to hold off rebalancing until you complete your task. Calling `SetAllocation("disable")` will disable allocation so Elasticsearch won't move/relocate any shards. Once you complete your task, calling `SetAllocation("enable")` will allow Elasticsearch to relocate shards again.

func (*Client) SetSetting

func (c *Client) SetSetting(setting string, value string) (string, string, error)

Set a new value for a cluster setting

Use case: You've doubled the number of nodes in your cluster and you want to increase the number of shards the cluster can relocate at one time. Calling `SetSetting("cluster.routing.allocation.cluster_concurrent_rebalance", "100")` will update that value with the cluster. Once data relocation is complete you can decrease the setting by calling `SetSetting("cluster.routing.allocation.cluster_concurrent_rebalance", "20")`.

func (*Client) VerifyRepository

func (c *Client) VerifyRepository(repository string) (bool, error)

Verify a snapshot repository

Use case: Have Elasticsearch verify a repository to make sure that all nodes can access the snapshot location correctly.

type ClusterHealth

type ClusterHealth struct {
	Cluster                string `json:"cluster"`
	Status                 string `json:"status"`
	RelocatingShards       int    `json:"relo,string"`
	InitializingShards     int    `json:"init,string"`
	UnassignedShards       int    `json:"unassign,string"`
	ActiveShardsPercentage string `json:"active_shards_percent"`
	Message                string
}

Holds information about the health of an Elasticsearch cluster, based on the _cat/health API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-health.html

type ClusterSetting

type ClusterSetting struct {
	Setting, Value string
}

A setting name and value with the setting name to be a "collapsed" version of the setting. A setting of:

{ "indices": { "recovery" : { "max_bytes_per_sec": "10mb" } } }

would be represented by:

ClusterSetting{ Setting: "indices.recovery.max_bytes_per_sec", Value: "10mb" }

type ClusterSettings

type ClusterSettings struct {
	PersistentSettings []ClusterSetting
	TransientSettings  []ClusterSetting
}

Holds slices for persistent and transient cluster settings.

type ExcludeSettings

type ExcludeSettings struct {
	Ips, Hosts, Names []string
}

Hold the values for what values are in the cluster.allocation.exclude settings. Relevant Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/allocation-filtering.html

type Index

type Index struct {
	Health        string `json:"health"`
	Status        string `json:"status"`
	Name          string `json:"index"`
	PrimaryShards int    `json:"pri,string"`
	ReplicaCount  int    `json:"rep,string"`
	IndexSize     string `json:"store.size"`
	DocumentCount int    `json:"docs.count,string"`
}

Holds information about an Elasticsearch index, based on the _cat/indices API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-indices.html

type Node

type Node struct {
	Name   string `json:"name"`
	Ip     string `json:"ip"`
	Id     string `json:"id"`
	Role   string `json:"role"`
	Master string `json:"master"`
}

Holds information about an Elasticsearch node, based on the _cat/nodes API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-nodes.html

type Snapshot

type Snapshot struct {
	State          string    `json:"state"`
	Name           string    `json:"snapshot"`
	StartTime      time.Time `json:"start_time,string"`
	EndTime        time.Time `json:"end_time,string"`
	DurationMillis int       `json:"duration_in_millis"`
	Indices        []string  `json:"indices"`
	Shards         struct {
		Total      int `json:"total"`
		Failed     int `json:"failed"`
		Successful int `json:"successful"`
	} `json:"shards"`
	Failures []struct {
		Index   string `json:"index"`
		ShardID int    `json:"shard_id"`
		Reason  string `json:"reason"`
		NodeID  string `json:"node_id"`
		Status  string `json:"status"`
	} `json:"failures"`
}

Holds information about an Elasticsearch snapshot, based on the snapshot API: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html