ingest

package
v8.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 17, 2025 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Simulate data ingestion. Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

Index

Constants

This section is empty.

Variables

View Source 
var ErrBuildPath = errors.New("cannot build path, check for missing path parameters")

ErrBuildPath is returned in case of missing parameters within the build of the request.

Functions

This section is empty.

Types

type Ingest 

type Ingest struct {
	// contains filtered or unexported fields
}

func New 

func New(tp elastictransport.Interface) *Ingest

Simulate data ingestion. Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-ingest-api.html

func (*Ingest) AddComponentTemplateSubstitution 

func (r *Ingest) AddComponentTemplateSubstitution(key string, value types.ComponentTemplateNodeVariant) *Ingest

func (*Ingest) AddIndexTemplateSubstitution 

func (r *Ingest) AddIndexTemplateSubstitution(key string, value types.IndexTemplateVariant) *Ingest

func (*Ingest) AddPipelineSubstitution 

func (r *Ingest) AddPipelineSubstitution(key string, value types.IngestPipelineVariant) *Ingest

func (*Ingest) ComponentTemplateSubstitutions 

func (r *Ingest) ComponentTemplateSubstitutions(componenttemplatesubstitutions map[string]types.ComponentTemplateNode) *Ingest

A map of component template names to substitute component template definition objects. API name: component_template_substitutions

func (Ingest) Do 

func (r Ingest) Do(providedCtx context.Context) (*Response, error)

Do runs the request through the transport, handle the response and returns a ingest.Response

func (*Ingest) Docs 

func (r *Ingest) Docs(docs ...types.DocumentVariant) *Ingest

Sample documents to test in the pipeline. API name: docs

func (*Ingest) ErrorTrace 

func (r *Ingest) ErrorTrace(errortrace bool) *Ingest

ErrorTrace When set to `true` Elasticsearch will include the full stack trace of errors when they occur. API name: error_trace

func (*Ingest) FilterPath 

func (r *Ingest) FilterPath(filterpaths ...string) *Ingest

FilterPath Comma-separated list of filters in dot notation which reduce the response returned by Elasticsearch. API name: filter_path

func (*Ingest) Header 

func (r *Ingest) Header(key, value string) *Ingest

Header set a key, value pair in the Ingest headers map.

func (*Ingest) HttpRequest 

func (r *Ingest) HttpRequest(ctx context.Context) (*http.Request, error)

HttpRequest returns the http.Request object built from the given parameters.

func (*Ingest) Human 

func (r *Ingest) Human(human bool) *Ingest

Human When set to `true` will return statistics in a format suitable for humans. For example `"exists_time": "1h"` for humans and `"eixsts_time_in_millis": 3600000` for computers. When disabled the human readable values will be omitted. This makes sense for responses being consumed only by machines. API name: human

func (*Ingest) Index 

func (r *Ingest) Index(index string) *Ingest

Index The index to simulate ingesting into. This value can be overridden by specifying an index on each document. If you specify this parameter in the request path, it is used for any documents that do not explicitly specify an index argument. API Name: index

func (*Ingest) IndexTemplateSubstitutions 

func (r *Ingest) IndexTemplateSubstitutions(indextemplatesubstitutions map[string]types.IndexTemplate) *Ingest

A map of index template names to substitute index template definition objects. API name: index_template_substitutions

func (*Ingest) MappingAddition 

func (r *Ingest) MappingAddition(mappingaddition types.TypeMappingVariant) *Ingest

API name: mapping_addition

func (Ingest) Perform 

func (r Ingest) Perform(providedCtx context.Context) (*http.Response, error)

Perform runs the http.Request through the provided transport and returns an http.Response.

func (*Ingest) Pipeline 

func (r *Ingest) Pipeline(pipelinename string) *Ingest

Pipeline The pipeline to use as the default pipeline. This value can be used to override the default pipeline of the index. API name: pipeline

func (*Ingest) PipelineSubstitutions 

func (r *Ingest) PipelineSubstitutions(pipelinesubstitutions map[string]types.IngestPipeline) *Ingest

Pipelines to test. If you don’t specify the `pipeline` request path parameter, this parameter is required. If you specify both this and the request path parameter, the API only uses the request path parameter. API name: pipeline_substitutions

func (*Ingest) Pretty 

func (r *Ingest) Pretty(pretty bool) *Ingest

Pretty If set to `true` the returned JSON will be "pretty-formatted". Only use this option for debugging only. API name: pretty

func (*Ingest) Raw 

func (r *Ingest) Raw(raw io.Reader) *Ingest

Raw takes a json payload as input which is then passed to the http.Request If specified Raw takes precedence on Request method.

func (*Ingest) Request 

func (r *Ingest) Request(req *Request) *Ingest

Request allows to set the request property with the appropriate payload.

type NewIngest 

type NewIngest func() *Ingest

NewIngest type alias for index.

func NewIngestFunc 

func NewIngestFunc(tp elastictransport.Interface) NewIngest

NewIngestFunc returns a new instance of Ingest with the provided transport. Used in the index of the library this allows to retrieve every apis in once place.

type Request 

type Request struct {

	// ComponentTemplateSubstitutions A map of component template names to substitute component template definition
	// objects.
	ComponentTemplateSubstitutions map[string]types.ComponentTemplateNode `json:"component_template_substitutions,omitempty"`
	// Docs Sample documents to test in the pipeline.
	Docs []types.Document `json:"docs"`
	// IndexTemplateSubstitutions A map of index template names to substitute index template definition
	// objects.
	IndexTemplateSubstitutions map[string]types.IndexTemplate `json:"index_template_substitutions,omitempty"`
	MappingAddition            *types.TypeMapping             `json:"mapping_addition,omitempty"`
	// PipelineSubstitutions Pipelines to test.
	// If you don’t specify the `pipeline` request path parameter, this parameter is
	// required.
	// If you specify both this and the request path parameter, the API only uses
	// the request path parameter.
	PipelineSubstitutions map[string]types.IngestPipeline `json:"pipeline_substitutions,omitempty"`
}

Request holds the request body struct for the package ingest

https://github.com/elastic/elasticsearch-specification/blob/f6a370d0fba975752c644fc730f7c45610e28f36/specification/simulate/ingest/SimulateIngestRequest.ts#L29-L100

func NewRequest 

func NewRequest() *Request

NewRequest returns a Request

func (*Request) FromJSON 

func (r *Request) FromJSON(data string) (*Request, error)

FromJSON allows to load an arbitrary json into the request structure

type Response 

type Response struct {
	Docs []types.SimulateIngestDocumentResult `json:"docs"`
}

Response holds the response body struct for the package ingest

https://github.com/elastic/elasticsearch-specification/blob/f6a370d0fba975752c644fc730f7c45610e28f36/specification/simulate/ingest/SimulateIngestResponse.ts#L27-L29

func NewResponse 

func NewResponse() *Response

NewResponse returns a Response

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.