elasticsearch

README

Elasticsearch adaptor

The elasticsearch adaptor sends data to defined endpoints. List of supported versions is below.

Version	Note
1.X	This version does not support bulk operations and will thus be much slower.
2.X	Will only receive bug fixes, please consider upgrading.
5.X	Most recent and supported version.

IMPORTANT

If you want to keep the source _id as the elasticsearch document _id, transporter will automatically do this. If you wish to use the auto-generated _id field for elasticsearch but would like to retain the originating _id from the source, you'll need to include a transform function similar to the following (assumes MongoDB source):

module.exports = function(msg) {
   msg.data["mongo_id"] = msg.data._id['$oid']
   msg.data = _.omit(msg.data, ["_id"]);
   return msg;
}

NOTE By using the elasticsearch auto-generated _id, it is not currently possible for transporter to process update/delete operations. Future work is planned in #39 to address this problem.

Configuration:

- es:
  type: elasticsearch
  uri: https://username:password@hostname:port/thisgetsignored
  timeout: 10s # optional, defaults to 30s
  aws_access_key: XXX # optional, used for signing requests to AWS Elasticsearch service
  aws_access_secret: XXX # optional, used for signing requests to AWS Elasticsearch service

Documentation

Index

• type AWSTransport
  • func (a AWSTransport) RoundTrip(req *http.Request) (*http.Response, error)
• type Config
• type Elasticsearch
  • func (e *Elasticsearch) Description() string
  • func (e *Elasticsearch) Listen() error
  • func (e *Elasticsearch) SampleConfig() string
  • func (e *Elasticsearch) Start() error
  • func (e *Elasticsearch) Stop() error

Types

type AWSTransport

type AWSTransport struct {
	Credentials awsauth.Credentials
	// contains filtered or unexported fields
}

AWSTransport handles wrapping requests to AWS Elasticsearch service

func (AWSTransport) RoundTrip

func (a AWSTransport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip implementation

type Config

type Config struct {
	URI             string `json:"uri" doc:"the uri to connect to, in the form mongodb://user:password@host.com:27017/auth_database"`
	Namespace       string `json:"namespace" doc:"mongo namespace to read/write"`
	Timeout         string `json:"timeout" doc:"timeout for establishing connection, format must be parsable by time.ParseDuration and defaults to 10s"`
	AWSAccessKeyID  string `json:"aws_access_key" doc:"credentials for use with AWS Elasticsearch service"`
	AWSAccessSecret string `json:"aws_access_secret" doc:"credentials for use with AWS Elasticsearch service"`
}

Config provides configuration options for an elasticsearch adaptor the notable difference between this and dbConfig is the presence of the Timeout option

type Elasticsearch

type Elasticsearch struct {
	// contains filtered or unexported fields
}

Elasticsearch is an adaptor to connect a pipeline to an elasticsearch cluster.

func (*Elasticsearch) Description

func (e *Elasticsearch) Description() string

Description for the Elasticsearcb adaptor

func (*Elasticsearch) Listen

func (e *Elasticsearch) Listen() error

Listen starts the listener

func (*Elasticsearch) SampleConfig

func (e *Elasticsearch) SampleConfig() string

SampleConfig for elasticsearch adaptor

func (*Elasticsearch) Start

func (e *Elasticsearch) Start() error

Start the adaptor as a source (not implemented)

func (*Elasticsearch) Stop

func (e *Elasticsearch) Stop() error

Stop the adaptor