elasticsearch

README

Elasticsearch adaptor

The elasticsearch adaptor sends data to defined endpoints. List of supported versions is below.

Version	Note
1.X	This version does not support bulk operations and will thus be much slower.
2.X	Will only receive bug fixes, please consider upgrading.
5.X	Most recent and supported version.

IMPORTANT

If you want to keep the source _id as the elasticsearch document _id, transporter will automatically do this. If you wish to use the auto-generated _id field for elasticsearch but would like to retain the originating _id from the source, you'll need to include a transform function similar to the following (assumes MongoDB source):

module.exports = function(msg) {
   msg.data["mongo_id"] = msg.data._id['$oid']
   msg.data = _.omit(msg.data, ["_id"]);
   return msg;
}

NOTE By using the elasticsearch auto-generated _id, it is not currently possible for transporter to process update/delete operations. Future work is planned in #39 to address this problem.

Configuration:

es = elasticsearch({
  "uri": "https://username:password@hostname:port/thisgetsignored"
  "timeout": "10s" // optional, defaults to 30s
  "aws_access_key": "XXX" // optional, used for signing requests to AWS Elasticsearch service
  "aws_access_secret": "XXX" // optional, used for signing requests to AWS Elasticsearch service
})

Documentation

Index

• Constants
• type AWSTransport
  • func (a AWSTransport) RoundTrip(req *http.Request) (*http.Response, error)
• type Elasticsearch
  • func (e *Elasticsearch) Client() (client.Client, error)
  • func (e *Elasticsearch) Description() string
  • func (e *Elasticsearch) Reader() (client.Reader, error)
  • func (e *Elasticsearch) SampleConfig() string
  • func (e *Elasticsearch) Writer(done chan struct{}, wg *sync.WaitGroup) (client.Writer, error)

Constants

const (
	// DefaultIndex is used when there is not one included in the provided URI.
	DefaultIndex = "test"
)

Types

type AWSTransport

type AWSTransport struct {
	Credentials awsauth.Credentials
	// contains filtered or unexported fields
}

AWSTransport handles wrapping requests to AWS Elasticsearch service

func (AWSTransport) RoundTrip

func (a AWSTransport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip implementation

type Elasticsearch

type Elasticsearch struct {
	adaptor.BaseConfig
	AWSAccessKeyID  string `json:"aws_access_key" doc:"credentials for use with AWS Elasticsearch service"`
	AWSAccessSecret string `json:"aws_access_secret" doc:"credentials for use with AWS Elasticsearch service"`
}

Elasticsearch is an adaptor to connect a pipeline to an elasticsearch cluster.

func (*Elasticsearch) Client

func (e *Elasticsearch) Client() (client.Client, error)

Client returns a client that doesn't do anything other than fulfill the client.Client interface.

func (*Elasticsearch) Description

func (e *Elasticsearch) Description() string

Description for the Elasticsearcb adaptor

func (*Elasticsearch) Reader

func (e *Elasticsearch) Reader() (client.Reader, error)

Reader returns an error because this adaptor is currently not supported as a Source.

func (*Elasticsearch) SampleConfig

func (e *Elasticsearch) SampleConfig() string

SampleConfig for elasticsearch adaptor

func (*Elasticsearch) Writer

func (e *Elasticsearch) Writer(done chan struct{}, wg *sync.WaitGroup) (client.Writer, error)

Writer determines the which underlying writer to used based on the cluster's version.