ingestnode

README

go-ingest-node

go-ingest-node provides a Go data model and JSON Schema for Elasticsearch Ingest Node pipelines. The model is generated from the elasticsearch-specification.

The Go types can be used for marshaling to/from JSON and YAML.

The JSON Schema file can be imported into popular editors and IDEs to provide validation and autocompletion when developing Elasticsearh ingest node pipelines.

Documentation

Overview

Package ingestnode provides a data model for Elasticsearch Ingest Node processors.

Package ingestnode provides a data model for Elasticsearch Ingest Node pipelines. The model is generated from the elasticsearch-specification.

Index

• type AppendProcessor
• type AttachmentProcessor
• type BytesProcessor
• type CircleProcessor
• type CommunityIDProcessor
• type ConvertProcessor
• type ConvertType
• type CsvProcessor
• type DateIndexNameProcessor
• type DateProcessor
• type DissectProcessor
• type DotExpanderProcessor
• type DropProcessor
• type EnrichProcessor
• type FailProcessor
• type Field
• type Fields
• type FingerprintDigest
• type FingerprintProcessor
• type ForeachProcessor
• type GeoGridProcessor
• type GeoGridTargetFormat
• type GeoGridTileType
• type GeoIpProcessor
• type GeoShapeRelation
• type GrokPattern
• type GrokProcessor
• type GsubProcessor
• type HtmlStripProcessor
• type Id
• type InferenceConfig
• type InferenceConfigClassification
• type InferenceConfigRegression
• type InferenceProcessor
• type InputConfig
• type IpLocationProcessor
• type JoinProcessor
• type JsonProcessor
• type JsonProcessorConflictStrategy
• type KeyValueProcessor
• type LowercaseProcessor
• type Metadata
• type Name
• type NetworkDirectionProcessor
• type Pipeline
• type PipelineProcessor
• type ProcessorBase
• type ProcessorContainer
• type RedactProcessor
• type RegisteredDomainProcessor
• type RemoveProcessor
• type RenameProcessor
• type RerouteProcessor
• type ScriptLanguage
• type ScriptProcessor
• type SetProcessor
• type SetSecurityUserProcessor
• type ShapeType
• type SortOrder
• type SortProcessor
• type SplitProcessor
• type TerminateProcessor
• type TrimProcessor
• type UppercaseProcessor
• type UriPartsProcessor
• type UrlDecodeProcessor
• type UserAgentProcessor
• type UserAgentProperty
• type VersionNumber

Types

type AppendProcessor

type AppendProcessor struct {
	ProcessorBase
	Field           Field `json:"field" yaml:"field"`                                                                     // The field to be appended to. Supports template snippets. Required.
	Value           any   `json:"value" yaml:"value"`                                                                     // The value to be appended. Supports template snippets. Required.
	AllowDuplicates *bool `json:"allow_duplicates,omitempty" jsonschema:"default=true" yaml:"allow_duplicates,omitempty"` // If `false`, the processor does not append values already present in the field.
}

Appends one or more values to an existing array if the field already exists and it is an array. Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar. Creates an array containing the provided values if the field doesn’t exist. Accepts a single value or an array of values.

Reference: append processor

type AttachmentProcessor

type AttachmentProcessor struct {
	ProcessorBase
	Field             Field    `json:"field" yaml:"field"`                                                                   // The field to get the base64 encoded field from. Required.
	IgnoreMissing     *bool    `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`  // If `true` and field does not exist, the processor quietly exits without modifying the document.
	IndexedChars      *int64   `json:"indexed_chars,omitempty" jsonschema:"default=100000" yaml:"indexed_chars,omitempty"`   // The number of chars being used for extraction to prevent huge fields. Use `-1` for no limit.
	IndexedCharsField *Field   `json:"indexed_chars_field,omitempty" yaml:"indexed_chars_field,omitempty"`                   // Field name from which you can overwrite the number of chars being used for extraction.
	Properties        []string `json:"properties,omitempty" yaml:"properties,omitempty"`                                     // Array of properties to select to be stored. Can be `content`, `title`, `name`, `author`, `keywords`, `date`, `content_type`, `content_length`, `language`.
	TargetField       *Field   `json:"target_field,omitempty" jsonschema:"default=attachment" yaml:"target_field,omitempty"` // The field that will hold the attachment information.
	RemoveBinary      *bool    `json:"remove_binary,omitempty" jsonschema:"default=false" yaml:"remove_binary,omitempty"`    // If true, the binary field will be removed from the document.
	ResourceName      *string  `json:"resource_name,omitempty" yaml:"resource_name,omitempty"`                               // Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.
}

The attachment processor lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library Tika.

Reference: attachment processor

type BytesProcessor

type BytesProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                  // The field to convert. Required.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	TargetField   *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`     // The field to assign the converted value to. By default, the field is updated in-place.
}

Converts a human readable byte value (for example `1kb`) to its value in bytes (for example `1024`). If the field is an array of strings, all members of the array will be converted. Supported human readable units are "b", "kb", "mb", "gb", "tb", "pb" case insensitive. An error will occur if the field is not a supported format or resultant value exceeds 2^63.

Reference: bytes processor

type CircleProcessor

type CircleProcessor struct {
	ProcessorBase
	ErrorDistance float64   `json:"error_distance" yaml:"error_distance"`                                                // The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for `geo_shape`, unit-less for `shape`). Required.
	Field         Field     `json:"field" yaml:"field"`                                                                  // The field to interpret as a circle. Either a string in WKT format or a map for GeoJSON. Required.
	IgnoreMissing *bool     `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	ShapeType     ShapeType `json:"shape_type" jsonschema:"enum=geo_shape,enum=shape" yaml:"shape_type"`                 // Which field mapping type is to be used when processing the circle: `geo_shape` or `shape`. Required.
	TargetField   *Field    `json:"target_field,omitempty" yaml:"target_field,omitempty"`                                // The field to assign the polygon shape to By default, the field is updated in-place.
}

Converts circle definitions of shapes to regular polygons which approximate them.

Reference: circle processor

type CommunityIDProcessor

type CommunityIDProcessor struct {
	ProcessorBase
	SourceIP        *Field `json:"source_ip,omitempty" jsonschema:"default=source.ip" yaml:"source_ip,omitempty"`                      // Field containing the source IP address.
	SourcePort      *Field `json:"source_port,omitempty" jsonschema:"default=source.port" yaml:"source_port,omitempty"`                // Field containing the source port.
	DestinationIP   *Field `json:"destination_ip,omitempty" jsonschema:"default=destination.ip" yaml:"destination_ip,omitempty"`       // Field containing the destination IP address.
	DestinationPort *Field `json:"destination_port,omitempty" jsonschema:"default=destination.port" yaml:"destination_port,omitempty"` // Field containing the destination port.
	IANANumber      *Field `json:"iana_number,omitempty" jsonschema:"default=network.iana_number" yaml:"iana_number,omitempty"`        // Field containing the IANA number.
	ICMPType        *Field `json:"icmp_type,omitempty" jsonschema:"default=icmp.type" yaml:"icmp_type,omitempty"`                      // Field containing the ICMP type.
	ICMPCode        *Field `json:"icmp_code,omitempty" jsonschema:"default=icmp.code" yaml:"icmp_code,omitempty"`                      // Field containing the ICMP code.
	Transport       *Field `json:"transport,omitempty" jsonschema:"default=network.transport" yaml:"transport,omitempty"`              // Field containing the transport protocol name or number. Used only when the iana_number field is not present. The following protocol names are currently supported: eigrp, gre, icmp, icmpv6, igmp, ipv6-icmp, ospf, pim, sctp, tcp, udp.
	TargetField     *Field `json:"target_field,omitempty" jsonschema:"default=network.community_id" yaml:"target_field,omitempty"`     // Output field for the community ID.
	Seed            *int32 `json:"seed,omitempty" jsonschema:"default=0" yaml:"seed,omitempty"`                                        // Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.
	IgnoreMissing   *bool  `json:"ignore_missing,omitempty" jsonschema:"default=true" yaml:"ignore_missing,omitempty"`                 // If true and any required fields are missing, the processor quietly exits without modifying the document.
}

Computes the Community ID for network flow data as defined in the Community ID Specification. You can use a community ID to correlate network events related to a single flow.

Reference: community_id processor

type ConvertProcessor

type ConvertProcessor struct {
	ProcessorBase
	Field         Field       `json:"field" yaml:"field"`                                                                                                     // The field whose value is to be converted. Required.
	IgnoreMissing *bool       `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`                                    // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	TargetField   *Field      `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`                                        // The field to assign the converted value to. By default, the `field` is updated in-place.
	Type          ConvertType `json:"type" jsonschema:"enum=integer,enum=long,enum=double,enum=float,enum=boolean,enum=ip,enum=string,enum=auto" yaml:"type"` // The type to convert the existing value to. Required.
}

Converts a field in the currently ingested document to a different type, such as converting a string to an integer. If the field value is an array, all members will be converted.

Reference: convert processor

type ConvertType

type ConvertType string

const (
	IntegerConvertType ConvertType = "integer"
	LongConvertType    ConvertType = "long"
	DoubleConvertType  ConvertType = "double"
	FloatConvertType   ConvertType = "float"
	BooleanConvertType ConvertType = "boolean"
	IPConvertType      ConvertType = "ip"
	StringConvertType  ConvertType = "string"
	AutoConvertType    ConvertType = "auto"
)

type CsvProcessor

type CsvProcessor struct {
	ProcessorBase
	EmptyValue    any     `json:"empty_value,omitempty" yaml:"empty_value,omitempty"`                      // Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes (`""`).
	Field         Field   `json:"field" yaml:"field"`                                                      // The field to extract data from. Required.
	IgnoreMissing *bool   `json:"ignore_missing,omitempty" yaml:"ignore_missing,omitempty"`                // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	Quote         *string `json:"quote,omitempty" jsonschema:"default=\"" yaml:"quote,omitempty"`          // Quote used in CSV, has to be single character string.
	Separator     *string `json:"separator,omitempty" jsonschema:"default=\\," yaml:"separator,omitempty"` // Separator used in CSV, has to be single character string.
	TargetFields  Fields  `json:"target_fields" yaml:"target_fields"`                                      // The array of fields to assign extracted values to. Required.
	Trim          *bool   `json:"trim,omitempty" yaml:"trim,omitempty"`                                    // Trim whitespaces in unquoted fields.
}

Extracts fields from CSV line out of a single text field within a document. Any empty field in CSV will be skipped.

Reference: csv processor

type DateIndexNameProcessor

type DateIndexNameProcessor struct {
	ProcessorBase
	DateFormats     []string `json:"date_formats,omitempty" yaml:"date_formats,omitempty"`                                           // An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
	DateRounding    string   `json:"date_rounding" yaml:"date_rounding"`                                                             // How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second). Supports template snippets. Required.
	Field           Field    `json:"field" yaml:"field"`                                                                             // The field to get the date or timestamp from. Required.
	IndexNameFormat *string  `json:"index_name_format,omitempty" jsonschema:"default=yyyy-MM-dd" yaml:"index_name_format,omitempty"` // The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.
	IndexNamePrefix *string  `json:"index_name_prefix,omitempty" yaml:"index_name_prefix,omitempty"`                                 // A prefix of the index name to be prepended before the printed date. Supports template snippets.
	Locale          *string  `json:"locale,omitempty" jsonschema:"default=ENGLISH" yaml:"locale,omitempty"`                          // The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
	Timezone        *string  `json:"timezone,omitempty" jsonschema:"default=UTC" yaml:"timezone,omitempty"`                          // The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
}

The purpose of this processor is to point documents to the right time based index based on a date or timestamp field in a document by using the date math index name support.

Reference: date_index_name processor

type DateProcessor

type DateProcessor struct {
	ProcessorBase
	Field        Field    `json:"field" yaml:"field"`                                                                                       // The field to get the date from. Required.
	Formats      []string `json:"formats" yaml:"formats"`                                                                                   // An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. Required.
	Locale       *string  `json:"locale,omitempty" jsonschema:"default=ENGLISH" yaml:"locale,omitempty"`                                    // The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.
	TargetField  *Field   `json:"target_field,omitempty" jsonschema:"default=@timestamp" yaml:"target_field,omitempty"`                     // The field that will hold the parsed date.
	Timezone     *string  `json:"timezone,omitempty" jsonschema:"default=UTC" yaml:"timezone,omitempty"`                                    // The timezone to use when parsing the date. Supports template snippets.
	OutputFormat *string  `json:"output_format,omitempty" jsonschema:"default=yyyy-MM-dd'T'HH:mm:ss.SSSXXX" yaml:"output_format,omitempty"` // The format to use when writing the date to target_field. Must be a valid java time pattern.
}

Parses dates from fields, and then uses the date or timestamp as the timestamp for the document.

Reference: date processor

type DissectProcessor

type DissectProcessor struct {
	ProcessorBase
	AppendSeparator *string `json:"append_separator,omitempty" yaml:"append_separator,omitempty"`                        // The character(s) that separate the appended fields.
	Field           Field   `json:"field" yaml:"field"`                                                                  // The field to dissect. Required.
	IgnoreMissing   *bool   `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	Pattern         string  `json:"pattern" yaml:"pattern"`                                                              // The pattern to apply to the field. Required.
}

Extracts structured fields out of a single text field by matching the text field against a delimiter-based pattern.

Reference: dissect processor

type DotExpanderProcessor

type DotExpanderProcessor struct {
	ProcessorBase
	Field    Field   `json:"field" yaml:"field"`                                                      // The field to expand into an object field. If set to `*`, all top-level fields will be expanded. Required.
	Override *bool   `json:"override,omitempty" jsonschema:"default=false" yaml:"override,omitempty"` // Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When `false`, the processor will merge conflicts by combining the old and the new values into an array. When `true`, the value from the expanded field will overwrite the existing value.
	Path     *string `json:"path,omitempty" yaml:"path,omitempty"`                                    // The field that contains the field to expand. Only required if the field to expand is part another object field, because the `field` option can only understand leaf fields.
}

Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.

Reference: dot_expander processor

type DropProcessor

type DropProcessor struct {
	ProcessorBase
}

Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition.

Reference: drop processor

type EnrichProcessor

type EnrichProcessor struct {
	ProcessorBase
	Field         Field             `json:"field" yaml:"field"`                                                                  // The field in the input document that matches the policies match_field used to retrieve the enrichment data. Supports template snippets. Required.
	IgnoreMissing *bool             `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	MaxMatches    *int32            `json:"max_matches,omitempty" jsonschema:"default=1" yaml:"max_matches,omitempty"`           // The maximum number of matched documents to include under the configured target field. The `target_field` will be turned into a json array if `max_matches` is higher than 1, otherwise `target_field` will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.
	Override      *bool             `json:"override,omitempty" jsonschema:"default=true" yaml:"override,omitempty"`              // If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
	PolicyName    string            `json:"policy_name" yaml:"policy_name"`                                                      // The name of the enrich policy to use. Required.
	ShapeRelation *GeoShapeRelation ``                                                                                           // A spatial relation operator used to match the geoshape of incoming documents to documents in the enrich index. This option is only used for `geo_match` enrich policy types.
	/* 151-byte string literal not displayed */
	TargetField Field `json:"target_field" yaml:"target_field"` // Field added to incoming documents to contain enrich data. This field contains both the `match_field` and `enrich_fields` specified in the enrich policy. Supports template snippets. Required.
}

The `enrich` processor can enrich documents with data from another index.

Reference: enrich processor

type FailProcessor

type FailProcessor struct {
	ProcessorBase
	Message string `json:"message" yaml:"message"` // The error message thrown by the processor. Supports template snippets. Required.
}

Raises an exception. This is useful for when you expect a pipeline to fail and want to relay a specific message to the requester.

Reference: fail processor

type Field

type Field string

type Fields

type Fields any

type FingerprintDigest

type FingerprintDigest string

const (
	Md5FingerprintDigest         FingerprintDigest = "MD5"
	Sha1FingerprintDigest        FingerprintDigest = "SHA-1"
	Sha256FingerprintDigest      FingerprintDigest = "SHA-256"
	Sha512FingerprintDigest      FingerprintDigest = "SHA-512"
	Murmurhash3FingerprintDigest FingerprintDigest = "MurmurHash3"
)

type FingerprintProcessor

type FingerprintProcessor struct {
	ProcessorBase
	Fields      Fields             `json:"fields" yaml:"fields"`                                                                  // Array of fields to include in the fingerprint. For objects, the processor hashes both the field key and value. For other fields, the processor hashes only the field value. Required.
	TargetField *Field             `json:"target_field,omitempty" jsonschema:"default=fingerprint" yaml:"target_field,omitempty"` // Output field for the fingerprint.
	Salt        *string            `json:"salt,omitempty" yaml:"salt,omitempty"`                                                  // Salt value for the hash function.
	Method      *FingerprintDigest ``                                                                                             // The hash method used to compute the fingerprint. Must be one of MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.
	/* 137-byte string literal not displayed */
	IgnoreMissing *bool `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.
}

Computes a hash of the document’s content. You can use this hash for content fingerprinting.

Reference: fingerprint processor

type ForeachProcessor

type ForeachProcessor struct {
	ProcessorBase
	Field         Field              `json:"field" yaml:"field"`                                                                  // Field containing array or object values. Required.
	IgnoreMissing *bool              `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true`, the processor silently exits without changing the document if the `field` is `null` or missing.
	Processor     ProcessorContainer `json:"processor" yaml:"processor"`                                                          // Ingest processor to run on each element. Required.
}

Runs an ingest processor on each element of an array or object.

Reference: foreach processor

type GeoGridProcessor

type GeoGridProcessor struct {
	ProcessorBase
	Field            string               `json:"field" yaml:"field"`                                                                                        // The field to interpret as a geo-tile.= The field format is determined by the `tile_type`. Required.
	TileType         GeoGridTileType      `json:"tile_type" jsonschema:"enum=geotile,enum=geohex,enum=geohash" yaml:"tile_type"`                             // Three tile formats are understood: geohash, geotile and geohex. Required.
	TargetField      *Field               `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`                           // The field to assign the polygon shape to, by default, the `field` is updated in-place.
	ParentField      *Field               `json:"parent_field,omitempty" yaml:"parent_field,omitempty"`                                                      // If specified and a parent tile exists, save that tile address to this field.
	ChildrenField    *Field               `json:"children_field,omitempty" yaml:"children_field,omitempty"`                                                  // If specified and children tiles exist, save those tile addresses to this field as an array of strings.
	NonChildrenField *Field               `json:"non_children_field,omitempty" yaml:"non_children_field,omitempty"`                                          // If specified and intersecting non-child tiles exist, save their addresses to this field as an array of strings.
	PrecisionField   *Field               `json:"precision_field,omitempty" yaml:"precision_field,omitempty"`                                                // If specified, save the tile precision (zoom) as an integer to this field.
	IgnoreMissing    *bool                `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`                       // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	TargetFormat     *GeoGridTargetFormat `json:"target_format,omitempty" jsonschema:"enum=geojson,enum=wkt,default=geojson" yaml:"target_format,omitempty"` // Which format to save the generated polygon in.
}

Converts geo-grid definitions of grid tiles or cells to regular bounding boxes or polygons which describe their shape. This is useful if there is a need to interact with the tile shapes as spatially indexable fields.

Reference: geo_grid processor

type GeoGridTargetFormat

type GeoGridTargetFormat string

const (
	GeojsonGeoGridTargetFormat GeoGridTargetFormat = "geojson"
	WktGeoGridTargetFormat     GeoGridTargetFormat = "wkt"
)

type GeoGridTileType

type GeoGridTileType string

const (
	GeotileGeoGridTileType GeoGridTileType = "geotile"
	GeohexGeoGridTileType  GeoGridTileType = "geohex"
	GeohashGeoGridTileType GeoGridTileType = "geohash"
)

type GeoIpProcessor

type GeoIpProcessor struct {
	ProcessorBase
	DatabaseFile                       *string  `json:"database_file,omitempty" jsonschema:"default=GeoLite2-City.mmdb" yaml:"database_file,omitempty"`           // The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.
	Field                              Field    `json:"field" yaml:"field"`                                                                                       // The field to get the ip address from for the geographical lookup. Required.
	FirstOnly                          *bool    `json:"first_only,omitempty" jsonschema:"default=true" yaml:"first_only,omitempty"`                               // If `true`, only the first found geoip data will be returned, even if the field contains an array.
	IgnoreMissing                      *bool    `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`                      // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	Properties                         []string `json:"properties,omitempty" yaml:"properties,omitempty"`                                                         // Controls what properties are added to the `target_field` based on the geoip lookup.
	TargetField                        *Field   `json:"target_field,omitempty" jsonschema:"default=geoip" yaml:"target_field,omitempty"`                          // The field that will hold the geographical information looked up from the MaxMind database.
	DownloadDatabaseOnPipelineCreation *bool    `json:"download_database_on_pipeline_creation,omitempty" yaml:"download_database_on_pipeline_creation,omitempty"` // If `true` (and if `ingest.geoip.downloader.eager.download` is `false`), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the `default_pipeline` or `final_pipeline` in an index.
}

The `geoip` processor adds information about the geographical location of an IPv4 or IPv6 address.

Reference: geoip processor

type GeoShapeRelation

type GeoShapeRelation string

const (
	IntersectsGeoShapeRelation GeoShapeRelation = "intersects"
	DisjointGeoShapeRelation   GeoShapeRelation = "disjoint"
	WithinGeoShapeRelation     GeoShapeRelation = "within"
	ContainsGeoShapeRelation   GeoShapeRelation = "contains"
)

type GrokPattern

type GrokPattern string

type GrokProcessor

type GrokProcessor struct {
	ProcessorBase
	EcsCompatibility   *string           `json:"ecs_compatibility,omitempty" jsonschema:"default=disabled" yaml:"ecs_compatibility,omitempty"` // Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.
	Field              Field             `json:"field" yaml:"field"`                                                                           // The field to use for grok expression parsing. Required.
	IgnoreMissing      *bool             `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`          // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	PatternDefinitions map[string]string `json:"pattern_definitions,omitempty" yaml:"pattern_definitions,omitempty"`                           // A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.
	Patterns           []GrokPattern     `json:"patterns" yaml:"patterns"`                                                                     // An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches. Required.
	TraceMatch         *bool             `json:"trace_match,omitempty" jsonschema:"default=false" yaml:"trace_match,omitempty"`                // When `true`, `_ingest._grok_match_index` will be inserted into your matched document’s metadata with the index into the pattern found in `patterns` that matched.
}

Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.

Reference: grok processor

type GsubProcessor

type GsubProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                  // The field to apply the replacement to. Required.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	Pattern       string `json:"pattern" yaml:"pattern"`                                                              // The pattern to be replaced. Required.
	Replacement   string `json:"replacement" yaml:"replacement"`                                                      // The string to replace the matching patterns with. Required.
	TargetField   *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`     // The field to assign the converted value to By default, the `field` is updated in-place.
}

Converts a string field by applying a regular expression and a replacement. If the field is an array of string, all members of the array will be converted. If any non-string values are encountered, the processor will throw an exception.

Reference: gsub processor

type HtmlStripProcessor

type HtmlStripProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                  // The string-valued field to remove HTML tags from. Required.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document,.
	TargetField   *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`     // The field to assign the converted value to By default, the `field` is updated in-place.
}

Removes HTML tags from the field. If the field is an array of strings, HTML tags will be removed from all members of the array.

type Id

type Id string

type InferenceConfig

type InferenceConfig struct {
	Regression     *InferenceConfigRegression     `json:"regression,omitempty" yaml:"regression,omitempty"`         // Regression configuration for inference.
	Classification *InferenceConfigClassification `json:"classification,omitempty" yaml:"classification,omitempty"` // Classification configuration for inference.
}

type InferenceConfigClassification

type InferenceConfigClassification struct {
	NumTopClasses                 *int32  `json:"num_top_classes,omitempty" jsonschema:"default=0" yaml:"num_top_classes,omitempty"`                                     // Specifies the number of top class predictions to return.
	NumTopFeatureImportanceValues *int32  `json:"num_top_feature_importance_values,omitempty" jsonschema:"default=0" yaml:"num_top_feature_importance_values,omitempty"` // Specifies the maximum number of feature importance values per document.
	ResultsField                  *Field  `json:"results_field,omitempty" jsonschema:"default=<dependent_variable>_prediction" yaml:"results_field,omitempty"`           // The field that is added to incoming documents to contain the inference prediction.
	TopClassesResultsField        *Field  `json:"top_classes_results_field,omitempty" jsonschema:"default=top_classes" yaml:"top_classes_results_field,omitempty"`       // Specifies the field to which the top classes are written.
	PredictionFieldType           *string `json:"prediction_field_type,omitempty" yaml:"prediction_field_type,omitempty"`                                                // Specifies the type of the predicted field to write. Valid values are: `string`, `number`, `boolean`.
}

type InferenceConfigRegression

type InferenceConfigRegression struct {
	ResultsField                  *Field `json:"results_field,omitempty" jsonschema:"default=<dependent_variable>_prediction" yaml:"results_field,omitempty"`           // The field that is added to incoming documents to contain the inference prediction.
	NumTopFeatureImportanceValues *int32 `json:"num_top_feature_importance_values,omitempty" jsonschema:"default=0" yaml:"num_top_feature_importance_values,omitempty"` // Specifies the maximum number of feature importance values per document.
}

type InferenceProcessor

type InferenceProcessor struct {
	ProcessorBase
	ModelID         Id               `json:"model_id" yaml:"model_id"`                                                                               // The ID or alias for the trained model, or the ID of the deployment. Required.
	TargetField     *Field           `json:"target_field,omitempty" jsonschema:"default=ml.inference.<processor_tag>" yaml:"target_field,omitempty"` // Field added to incoming documents to contain results objects.
	FieldMap        map[string]any   `json:"field_map,omitempty" yaml:"field_map,omitempty"`                                                         // Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.
	InferenceConfig *InferenceConfig `json:"inference_config,omitempty" yaml:"inference_config,omitempty"`                                           // Contains the inference type and its options.
	InputOutput     any              `json:"input_output,omitempty" yaml:"input_output,omitempty"`                                                   // Input fields for inference and output (destination) fields for the inference results. This option is incompatible with the target_field and field_map options.
	IgnoreMissing   *bool            `json:"ignore_missing,omitempty" yaml:"ignore_missing,omitempty"`                                               // If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.
}

Uses a pre-trained data frame analytics model or a model deployed for natural language processing tasks to infer against the data that is being ingested in the pipeline.

Reference: inference processor

type InputConfig

type InputConfig struct {
	InputField  string `json:"input_field" yaml:"input_field"`
	OutputField string `json:"output_field" yaml:"output_field"`
}

type IpLocationProcessor

type IpLocationProcessor struct {
	ProcessorBase
	DatabaseFile                       *string  `json:"database_file,omitempty" jsonschema:"default=GeoLite2-City.mmdb" yaml:"database_file,omitempty"`           // The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.
	Field                              Field    `json:"field" yaml:"field"`                                                                                       // The field to get the ip address from for the geographical lookup. Required.
	FirstOnly                          *bool    `json:"first_only,omitempty" jsonschema:"default=true" yaml:"first_only,omitempty"`                               // If `true`, only the first found IP location data will be returned, even if the field contains an array.
	IgnoreMissing                      *bool    `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`                      // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	Properties                         []string `json:"properties,omitempty" yaml:"properties,omitempty"`                                                         // Controls what properties are added to the `target_field` based on the IP location lookup.
	TargetField                        *Field   `json:"target_field,omitempty" jsonschema:"default=geoip" yaml:"target_field,omitempty"`                          // The field that will hold the geographical information looked up from the MaxMind database.
	DownloadDatabaseOnPipelineCreation *bool    `json:"download_database_on_pipeline_creation,omitempty" yaml:"download_database_on_pipeline_creation,omitempty"` // If `true` (and if `ingest.geoip.downloader.eager.download` is `false`), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the `default_pipeline` or `final_pipeline` in an index.
}

Currently an undocumented alias for GeoIP Processor.

type JoinProcessor

type JoinProcessor struct {
	ProcessorBase
	Field       Field  `json:"field" yaml:"field"`                                                              // Field containing array values to join. Required.
	Separator   string `json:"separator" yaml:"separator"`                                                      // The separator character. Required.
	TargetField *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"` // The field to assign the joined value to. By default, the field is updated in-place.
}

Joins each element of an array into a single string using a separator character between each element. Throws an error when the field is not an array.

Reference: join processor

type JsonProcessor

type JsonProcessor struct {
	ProcessorBase
	AddToRoot                 *bool                          `json:"add_to_root,omitempty" jsonschema:"default=false" yaml:"add_to_root,omitempty"` // Flag that forces the parsed JSON to be added at the top level of the document. `target_field` must not be set when this option is chosen.
	AddToRootConflictStrategy *JsonProcessorConflictStrategy ``                                                                                     // When set to `replace`, root fields that conflict with fields from the parsed JSON will be overridden. When set to `merge`, conflicting fields will be merged. Only applicable `if add_to_root` is set to true.
	/* 130-byte string literal not displayed */
	AllowDuplicateKeys *bool  `json:"allow_duplicate_keys,omitempty" jsonschema:"default=false" yaml:"allow_duplicate_keys,omitempty"` // When set to `true`, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.
	Field              Field  `json:"field" yaml:"field"`                                                                              // The field to be parsed. Required.
	TargetField        *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`                 // The field that the converted structured object will be written into. Any existing content in this field will be overwritten.
}

Converts a JSON string into a structured JSON object.

Reference: json processor

type JsonProcessorConflictStrategy

type JsonProcessorConflictStrategy string

const (
	ReplaceJSONProcessorConflictStrategy JsonProcessorConflictStrategy = "replace"
	MergeJSONProcessorConflictStrategy   JsonProcessorConflictStrategy = "merge"
)

type KeyValueProcessor

type KeyValueProcessor struct {
	ProcessorBase
	ExcludeKeys   []string `json:"exclude_keys,omitempty" yaml:"exclude_keys,omitempty"`                                // List of keys to exclude from document.
	Field         Field    `json:"field" yaml:"field"`                                                                  // The field to be parsed. Supports template snippets. Required.
	FieldSplit    string   `json:"field_split" yaml:"field_split"`                                                      // Regex pattern to use for splitting key-value pairs. Required.
	IgnoreMissing *bool    `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	IncludeKeys   []string `json:"include_keys,omitempty" yaml:"include_keys,omitempty"`                                // List of keys to filter and insert into document. Defaults to including all keys.
	Prefix        *string  `json:"prefix,omitempty" yaml:"prefix,omitempty"`                                            // Prefix to be added to extracted keys.
	StripBrackets *bool    `json:"strip_brackets,omitempty" jsonschema:"default=false" yaml:"strip_brackets,omitempty"` // If `true`. strip brackets `()`, `<>`, `[]` as well as quotes `'` and `"` from extracted values.
	TargetField   *Field   `json:"target_field,omitempty" yaml:"target_field,omitempty"`                                // The field to insert the extracted keys into. Defaults to the root of the document. Supports template snippets.
	TrimKey       *string  `json:"trim_key,omitempty" yaml:"trim_key,omitempty"`                                        // String of characters to trim from extracted keys.
	TrimValue     *string  `json:"trim_value,omitempty" yaml:"trim_value,omitempty"`                                    // String of characters to trim from extracted values.
	ValueSplit    string   `json:"value_split" yaml:"value_split"`                                                      // Regex pattern to use for splitting the key from the value within a key-value pair. Required.
}

This processor helps automatically parse messages (or specific event fields) which are of the `foo=bar` variety.

Reference: kv processor

type LowercaseProcessor

type LowercaseProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                  // The field to make lowercase. Required.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	TargetField   *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`     // The field to assign the converted value to. By default, the field is updated in-place.
}

Converts a string to its lowercase equivalent. If the field is an array of strings, all members of the array will be converted.

Reference: lowercase processor

type Metadata

type Metadata map[string]any

type Name

type Name string

type NetworkDirectionProcessor

type NetworkDirectionProcessor struct {
	ProcessorBase
	SourceIP              *Field   `json:"source_ip,omitempty" jsonschema:"default=source.ip" yaml:"source_ip,omitempty"`                // Field containing the source IP address.
	DestinationIP         *Field   `json:"destination_ip,omitempty" jsonschema:"default=destination.ip" yaml:"destination_ip,omitempty"` // Field containing the destination IP address.
	TargetField           *Field   `json:"target_field,omitempty" jsonschema:"default=network.direction" yaml:"target_field,omitempty"`  // Output field for the network direction.
	InternalNetworks      []string `json:"internal_networks,omitempty" yaml:"internal_networks,omitempty"`                               // List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.
	InternalNetworksField *Field   `json:"internal_networks_field,omitempty" yaml:"internal_networks_field,omitempty"`                   // A field on the given document to read the internal_networks configuration from.
	IgnoreMissing         *bool    `json:"ignore_missing,omitempty" jsonschema:"default=true" yaml:"ignore_missing,omitempty"`           // If true and any required fields are missing, the processor quietly exits without modifying the document.
}

Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.

Reference: network_direction processor

type Pipeline

type Pipeline struct {
	Description *string              `json:"description,omitempty" yaml:"description,omitempty"`                          // Description of the ingest pipeline.
	OnFailure   []ProcessorContainer `json:"on_failure,omitempty" yaml:"on_failure,omitempty"`                            // Processors to run immediately after a processor failure.
	Processors  []ProcessorContainer `json:"processors,omitempty" yaml:"processors,omitempty"`                            // Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.
	Version     *VersionNumber       `json:"version,omitempty" yaml:"version,omitempty"`                                  // Version number used by external systems to track ingest pipelines.
	Deprecated  *bool                `json:"deprecated,omitempty" jsonschema:"default=false" yaml:"deprecated,omitempty"` // Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.
	Meta        *Metadata            `json:"_meta,omitempty" yaml:"_meta,omitempty"`                                      // Arbitrary metadata about the ingest pipeline. This map is not automatically generated by Elasticsearch.
}

type PipelineProcessor

type PipelineProcessor struct {
	ProcessorBase
	Name                  Name  `json:"name" yaml:"name"`                                                                                      // The name of the pipeline to execute. Supports template snippets. Required.
	IgnoreMissingPipeline *bool `json:"ignore_missing_pipeline,omitempty" jsonschema:"default=false" yaml:"ignore_missing_pipeline,omitempty"` // Whether to ignore missing pipelines instead of failing.
}

Executes another pipeline.

Reference: pipeline processor

type ProcessorBase

type ProcessorBase struct {
	Description   *string              `json:"description,omitempty" yaml:"description,omitempty"`       // Description of the processor. Useful for describing the purpose of the processor or its configuration.
	If            *string              `json:"if,omitempty" yaml:"if,omitempty"`                         // Conditionally execute the processor.
	IgnoreFailure *bool                `json:"ignore_failure,omitempty" yaml:"ignore_failure,omitempty"` // Ignore failures for the processor.
	OnFailure     []ProcessorContainer `json:"on_failure,omitempty" yaml:"on_failure,omitempty"`         // Handle failures for the processor.
	Tag           *string              `json:"tag,omitempty" yaml:"tag,omitempty"`                       // Identifier for the processor. Useful for debugging and metrics.
}

type ProcessorContainer

type ProcessorContainer struct {
	Append           *AppendProcessor           `json:"append,omitempty" yaml:"append,omitempty"`                       // Appends one or more values to an existing array if the field already exists and it is an array. Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar. Creates an array containing the provided values if the field doesn’t exist. Accepts a single value or an array of values.
	Attachment       *AttachmentProcessor       `json:"attachment,omitempty" yaml:"attachment,omitempty"`               // The attachment processor lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library Tika.
	Bytes            *BytesProcessor            `json:"bytes,omitempty" yaml:"bytes,omitempty"`                         // Converts a human readable byte value (for example `1kb`) to its value in bytes (for example `1024`). If the field is an array of strings, all members of the array will be converted. Supported human readable units are "b", "kb", "mb", "gb", "tb", "pb" case insensitive. An error will occur if the field is not a supported format or resultant value exceeds 2^63.
	Circle           *CircleProcessor           `json:"circle,omitempty" yaml:"circle,omitempty"`                       // Converts circle definitions of shapes to regular polygons which approximate them.
	CommunityID      *CommunityIDProcessor      `json:"community_id,omitempty" yaml:"community_id,omitempty"`           // Computes the Community ID for network flow data as defined in the Community ID Specification. You can use a community ID to correlate network events related to a single flow.
	Convert          *ConvertProcessor          `json:"convert,omitempty" yaml:"convert,omitempty"`                     // Converts a field in the currently ingested document to a different type, such as converting a string to an integer. If the field value is an array, all members will be converted.
	CSV              *CsvProcessor              `json:"csv,omitempty" yaml:"csv,omitempty"`                             // Extracts fields from CSV line out of a single text field within a document. Any empty field in CSV will be skipped.
	Date             *DateProcessor             `json:"date,omitempty" yaml:"date,omitempty"`                           // Parses dates from fields, and then uses the date or timestamp as the timestamp for the document.
	DateIndexName    *DateIndexNameProcessor    `json:"date_index_name,omitempty" yaml:"date_index_name,omitempty"`     // The purpose of this processor is to point documents to the right time based index based on a date or timestamp field in a document by using the date math index name support.
	Dissect          *DissectProcessor          `json:"dissect,omitempty" yaml:"dissect,omitempty"`                     // Extracts structured fields out of a single text field by matching the text field against a delimiter-based pattern.
	DotExpander      *DotExpanderProcessor      `json:"dot_expander,omitempty" yaml:"dot_expander,omitempty"`           // Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.
	Drop             *DropProcessor             `json:"drop,omitempty" yaml:"drop,omitempty"`                           // Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition.
	Enrich           *EnrichProcessor           `json:"enrich,omitempty" yaml:"enrich,omitempty"`                       // The `enrich` processor can enrich documents with data from another index.
	Fail             *FailProcessor             `json:"fail,omitempty" yaml:"fail,omitempty"`                           // Raises an exception. This is useful for when you expect a pipeline to fail and want to relay a specific message to the requester.
	Fingerprint      *FingerprintProcessor      `json:"fingerprint,omitempty" yaml:"fingerprint,omitempty"`             // Computes a hash of the document’s content. You can use this hash for content fingerprinting.
	Foreach          *ForeachProcessor          `json:"foreach,omitempty" yaml:"foreach,omitempty"`                     // Runs an ingest processor on each element of an array or object.
	IPLocation       *IpLocationProcessor       `json:"ip_location,omitempty" yaml:"ip_location,omitempty"`             // Currently an undocumented alias for GeoIP Processor.
	GeoGrid          *GeoGridProcessor          `json:"geo_grid,omitempty" yaml:"geo_grid,omitempty"`                   // Converts geo-grid definitions of grid tiles or cells to regular bounding boxes or polygons which describe their shape. This is useful if there is a need to interact with the tile shapes as spatially indexable fields.
	Geoip            *GeoIpProcessor            `json:"geoip,omitempty" yaml:"geoip,omitempty"`                         // The `geoip` processor adds information about the geographical location of an IPv4 or IPv6 address.
	Grok             *GrokProcessor             `json:"grok,omitempty" yaml:"grok,omitempty"`                           // Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.
	Gsub             *GsubProcessor             `json:"gsub,omitempty" yaml:"gsub,omitempty"`                           // Converts a string field by applying a regular expression and a replacement. If the field is an array of string, all members of the array will be converted. If any non-string values are encountered, the processor will throw an exception.
	HtmlStrip        *HtmlStripProcessor        `json:"html_strip,omitempty" yaml:"html_strip,omitempty"`               // Removes HTML tags from the field. If the field is an array of strings, HTML tags will be removed from all members of the array.
	Inference        *InferenceProcessor        `json:"inference,omitempty" yaml:"inference,omitempty"`                 // Uses a pre-trained data frame analytics model or a model deployed for natural language processing tasks to infer against the data that is being ingested in the pipeline.
	Join             *JoinProcessor             `json:"join,omitempty" yaml:"join,omitempty"`                           // Joins each element of an array into a single string using a separator character between each element. Throws an error when the field is not an array.
	JSON             *JsonProcessor             `json:"json,omitempty" yaml:"json,omitempty"`                           // Converts a JSON string into a structured JSON object.
	KV               *KeyValueProcessor         `json:"kv,omitempty" yaml:"kv,omitempty"`                               // This processor helps automatically parse messages (or specific event fields) which are of the `foo=bar` variety.
	Lowercase        *LowercaseProcessor        `json:"lowercase,omitempty" yaml:"lowercase,omitempty"`                 // Converts a string to its lowercase equivalent. If the field is an array of strings, all members of the array will be converted.
	NetworkDirection *NetworkDirectionProcessor `json:"network_direction,omitempty" yaml:"network_direction,omitempty"` // Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.
	Pipeline         *PipelineProcessor         `json:"pipeline,omitempty" yaml:"pipeline,omitempty"`                   // Executes another pipeline.
	Redact           *RedactProcessor           `json:"redact,omitempty" yaml:"redact,omitempty"`                       // The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns. The processor can be used to obscure Personal Identifying Information (PII) by configuring it to detect known patterns such as email or IP addresses. Text that matches a Grok pattern is replaced with a configurable string such as `<EMAIL>` where an email address is matched or simply replace all matches with the text `<REDACTED>` if preferred.
	RegisteredDomain *RegisteredDomainProcessor `json:"registered_domain,omitempty" yaml:"registered_domain,omitempty"` // Extracts the registered domain (also known as the effective top-level domain or eTLD), sub-domain, and top-level domain from a fully qualified domain name (FQDN). Uses the registered domains defined in the Mozilla Public Suffix List.
	Remove           *RemoveProcessor           `json:"remove,omitempty" yaml:"remove,omitempty"`                       // Removes existing fields. If one field doesn’t exist, an exception will be thrown.
	Rename           *RenameProcessor           `json:"rename,omitempty" yaml:"rename,omitempty"`                       // Renames an existing field. If the field doesn’t exist or the new name is already used, an exception will be thrown.
	Reroute          *RerouteProcessor          `json:"reroute,omitempty" yaml:"reroute,omitempty"`                     // Routes a document to another target index or data stream. When setting the `destination` option, the target is explicitly specified and the dataset and namespace options can’t be set. When the `destination` option is not set, this processor is in a data stream mode. Note that in this mode, the reroute processor can only be used on data streams that follow the data stream naming scheme.
	Script           *ScriptProcessor           `json:"script,omitempty" yaml:"script,omitempty"`                       // Runs an inline or stored script on incoming documents. The script runs in the `ingest` context.
	Set              *SetProcessor              `json:"set,omitempty" yaml:"set,omitempty"`                             // Adds a field with the specified value. If the field already exists, its value will be replaced with the provided one.
	SetSecurityUser  *SetSecurityUserProcessor  `json:"set_security_user,omitempty" yaml:"set_security_user,omitempty"` // Sets user-related details (such as `username`, `roles`, `email`, `full_name`, `metadata`, `api_key`, `realm` and `authentication_type`) from the current authenticated user to the current document by pre-processing the ingest.
	Sort             *SortProcessor             `json:"sort,omitempty" yaml:"sort,omitempty"`                           // Sorts the elements of an array ascending or descending. Homogeneous arrays of numbers will be sorted numerically, while arrays of strings or heterogeneous arrays of strings + numbers will be sorted lexicographically. Throws an error when the field is not an array.
	Split            *SplitProcessor            `json:"split,omitempty" yaml:"split,omitempty"`                         // Splits a field into an array using a separator character. Only works on string fields.
	Terminate        *TerminateProcessor        `json:"terminate,omitempty" yaml:"terminate,omitempty"`                 // Terminates the current ingest pipeline, causing no further processors to be run. This will normally be executed conditionally, using the `if` option.
	Trim             *TrimProcessor             `json:"trim,omitempty" yaml:"trim,omitempty"`                           // Trims whitespace from a field. If the field is an array of strings, all members of the array will be trimmed. This only works on leading and trailing whitespace.
	Uppercase        *UppercaseProcessor        `json:"uppercase,omitempty" yaml:"uppercase,omitempty"`                 // Converts a string to its uppercase equivalent. If the field is an array of strings, all members of the array will be converted.
	URLDecode        *UrlDecodeProcessor        `json:"urldecode,omitempty" yaml:"urldecode,omitempty"`                 // URL-decodes a string. If the field is an array of strings, all members of the array will be decoded.
	URIParts         *UriPartsProcessor         `json:"uri_parts,omitempty" yaml:"uri_parts,omitempty"`                 // Parses a Uniform Resource Identifier (URI) string and extracts its components as an object. This URI object includes properties for the URI’s domain, path, fragment, port, query, scheme, user info, username, and password.
	UserAgent        *UserAgentProcessor        `json:"user_agent,omitempty" yaml:"user_agent,omitempty"`               // The `user_agent` processor extracts details from the user agent string a browser sends with its web requests. This processor adds this information by default under the `user_agent` field.
}

type RedactProcessor

type RedactProcessor struct {
	ProcessorBase
	Field              Field             `json:"field" yaml:"field"`       // The field to be redacted. Required.
	Patterns           []GrokPattern     `json:"patterns" yaml:"patterns"` // A list of grok expressions to match and redact named captures with. Required.
	PatternDefinitions map[string]string `json:"pattern_definitions,omitempty" yaml:"pattern_definitions,omitempty"`
	Prefix             *string           `json:"prefix,omitempty" jsonschema:"default=<" yaml:"prefix,omitempty"`                             // Start a redacted section with this token.
	Suffix             *string           `json:"suffix,omitempty" jsonschema:"default=>" yaml:"suffix,omitempty"`                             // End a redacted section with this token.
	IgnoreMissing      *bool             `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`         // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	SkipIfUnlicensed   *bool             `json:"skip_if_unlicensed,omitempty" jsonschema:"default=false" yaml:"skip_if_unlicensed,omitempty"` // If `true` and the current license does not support running redact processors, then the processor quietly exits without modifying the document.
	TraceRedact        *bool             `json:"trace_redact,omitempty" jsonschema:"default=false" yaml:"trace_redact,omitempty"`             // If `true` then ingest metadata `_ingest._redact._is_redacted` is set to `true` if the document has been redacted.
}

The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns. The processor can be used to obscure Personal Identifying Information (PII) by configuring it to detect known patterns such as email or IP addresses. Text that matches a Grok pattern is replaced with a configurable string such as `<EMAIL>` where an email address is matched or simply replace all matches with the text `<REDACTED>` if preferred.

Reference: redact processor

type RegisteredDomainProcessor

type RegisteredDomainProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                 // Field containing the source FQDN. Required.
	TargetField   *Field `json:"target_field,omitempty" yaml:"target_field,omitempty"`                               // Object field containing extracted domain components. If an empty string, the processor adds components to the document’s root.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=true" yaml:"ignore_missing,omitempty"` // If true and any required fields are missing, the processor quietly exits without modifying the document.
}

Extracts the registered domain (also known as the effective top-level domain or eTLD), sub-domain, and top-level domain from a fully qualified domain name (FQDN). Uses the registered domains defined in the Mozilla Public Suffix List.

Reference: registered_domain processor

type RemoveProcessor

type RemoveProcessor struct {
	ProcessorBase
	Field         Fields  `json:"field" yaml:"field"`                                                                  // Fields to be removed. Supports template snippets. Required.
	Keep          *Fields `json:"keep,omitempty" yaml:"keep,omitempty"`                                                // Fields to be kept. When set, all fields other than those specified are removed.
	IgnoreMissing *bool   `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
}

Removes existing fields. If one field doesn’t exist, an exception will be thrown.

Reference: remove processor

type RenameProcessor

type RenameProcessor struct {
	ProcessorBase
	Field         Field `json:"field" yaml:"field"`                                                                  // The field to be renamed. Supports template snippets. Required.
	IgnoreMissing *bool `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	TargetField   Field `json:"target_field" yaml:"target_field"`                                                    // The new name of the field. Supports template snippets. Required.
}

Renames an existing field. If the field doesn’t exist or the new name is already used, an exception will be thrown.

Reference: rename processor

type RerouteProcessor

type RerouteProcessor struct {
	ProcessorBase
	Destination *string `json:"destination,omitempty" yaml:"destination,omitempty"` // A static value for the target. Can’t be set when the dataset or namespace option is set.
	Dataset     any     `json:"dataset,omitempty" yaml:"dataset,omitempty"`         // Field references or a static value for the dataset part of the data stream name. In addition to the criteria for index names, cannot contain - and must be no longer than 100 characters. Example values are nginx.access and nginx.error.  Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the <dataset> part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.  default {{data_stream.dataset}}.
	Namespace   any     `json:"namespace,omitempty" yaml:"namespace,omitempty"`     // Field references or a static value for the namespace part of the data stream name. See the criteria for index names for allowed characters. Must be no longer than 100 characters.  Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the <namespace> part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.  default {{data_stream.namespace}}.
}

Routes a document to another target index or data stream. When setting the `destination` option, the target is explicitly specified and the dataset and namespace options can’t be set. When the `destination` option is not set, this processor is in a data stream mode. Note that in this mode, the reroute processor can only be used on data streams that follow the data stream naming scheme.

Reference: reroute processor

type ScriptLanguage

type ScriptLanguage string

const (
	PainlessScriptLanguage   ScriptLanguage = "painless"
	ExpressionScriptLanguage ScriptLanguage = "expression"
	MustacheScriptLanguage   ScriptLanguage = "mustache"
	JavaScriptLanguage       ScriptLanguage = "java"
)

type ScriptProcessor

type ScriptProcessor struct {
	ProcessorBase
	ID   *Id             `json:"id,omitempty" yaml:"id,omitempty"` // ID of a stored script. If no `source` is specified, this parameter is required.
	Lang *ScriptLanguage ``                                        // Script language.
	/* 127-byte string literal not displayed */
	Params map[string]any `json:"params,omitempty" yaml:"params,omitempty"` // Object containing parameters for the script.
	Source *string        `json:"source,omitempty" yaml:"source,omitempty"` // Inline script. If no `id` is specified, this parameter is required.
}

Runs an inline or stored script on incoming documents. The script runs in the `ingest` context.

Reference: script processor

type SetProcessor

type SetProcessor struct {
	ProcessorBase
	CopyFrom         *Field  `json:"copy_from,omitempty" yaml:"copy_from,omitempty"`                                              // The origin field which will be copied to `field`, cannot set `value` simultaneously. Supported data types are `boolean`, `number`, `array`, `object`, `string`, `date`, etc.
	Field            Field   `json:"field" yaml:"field"`                                                                          // The field to insert, upsert, or update. Supports template snippets. Required.
	IgnoreEmptyValue *bool   `json:"ignore_empty_value,omitempty" jsonschema:"default=false" yaml:"ignore_empty_value,omitempty"` // If `true` and `value` is a template snippet that evaluates to `null` or the empty string, the processor quietly exits without modifying the document.
	MediaType        *string `json:"media_type,omitempty" yaml:"media_type,omitempty"`                                            // The media type for encoding `value`. Applies only when value is a template snippet. Must be one of `application/json`, `text/plain`, or `application/x-www-form-urlencoded`.
	Override         *bool   `json:"override,omitempty" jsonschema:"default=true" yaml:"override,omitempty"`                      // If `true` processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
	Value            any     `json:"value,omitempty" yaml:"value,omitempty"`                                                      // The value to be set for the field. Supports template snippets. May specify only one of `value` or `copy_from`.
}

Adds a field with the specified value. If the field already exists, its value will be replaced with the provided one.

Reference: set processor

type SetSecurityUserProcessor

type SetSecurityUserProcessor struct {
	ProcessorBase
	Field      Field    `json:"field" yaml:"field"`                               // The field to store the user information into. Required.
	Properties []string `json:"properties,omitempty" yaml:"properties,omitempty"` // Controls what user related properties are added to the field.
}

Sets user-related details (such as `username`, `roles`, `email`, `full_name`, `metadata`, `api_key`, `realm` and `authentication_type`) from the current authenticated user to the current document by pre-processing the ingest.

Reference: set_security_user processor

type ShapeType

type ShapeType string

const (
	GeoShapeShapeType ShapeType = "geo_shape"
	ShapeShapeType    ShapeType = "shape"
)

type SortOrder

type SortOrder string

const (
	AscSortOrder  SortOrder = "asc"
	DescSortOrder SortOrder = "desc"
)

type SortProcessor

type SortProcessor struct {
	ProcessorBase
	Field       Field      `json:"field" yaml:"field"`                                                                 // The field to be sorted. Required.
	Order       *SortOrder `json:"order,omitempty" jsonschema:"enum=asc,enum=desc,default=asc" yaml:"order,omitempty"` // The sort order to use. Accepts `"asc"` or `"desc"`.
	TargetField *Field     `json:"target_field,omitempty" yaml:"target_field,omitempty"`                               // The field to assign the sorted value to. By default, the field is updated in-place.
}

Sorts the elements of an array ascending or descending. Homogeneous arrays of numbers will be sorted numerically, while arrays of strings or heterogeneous arrays of strings + numbers will be sorted lexicographically. Throws an error when the field is not an array.

Reference: sort processor

type SplitProcessor

type SplitProcessor struct {
	ProcessorBase
	Field            Field  `json:"field" yaml:"field"`                                                                        // The field to split. Required.
	IgnoreMissing    *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`       // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	PreserveTrailing *bool  `json:"preserve_trailing,omitempty" jsonschema:"default=false" yaml:"preserve_trailing,omitempty"` // Preserves empty trailing fields, if any.
	Separator        string `json:"separator" yaml:"separator"`                                                                // A regex which matches the separator, for example, `,` or `\s+`. Required.
	TargetField      *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`           // The field to assign the split value to. By default, the field is updated in-place.
}

Splits a field into an array using a separator character. Only works on string fields.

Reference: split processor

type TerminateProcessor

type TerminateProcessor struct {
	ProcessorBase
}

Terminates the current ingest pipeline, causing no further processors to be run. This will normally be executed conditionally, using the `if` option.

Reference: terminate processor

type TrimProcessor

type TrimProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                  // The string-valued field to trim whitespace from. Required.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	TargetField   *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`     // The field to assign the trimmed value to. By default, the field is updated in-place.
}

Trims whitespace from a field. If the field is an array of strings, all members of the array will be trimmed. This only works on leading and trailing whitespace.

Reference: trim processor

type UppercaseProcessor

type UppercaseProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                  // The field to make uppercase. Required.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	TargetField   *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`     // The field to assign the converted value to. By default, the field is updated in-place.
}

Converts a string to its uppercase equivalent. If the field is an array of strings, all members of the array will be converted.

Reference: uppercase processor

type UriPartsProcessor

type UriPartsProcessor struct {
	ProcessorBase
	Field              Field  `json:"field" yaml:"field"`                                                                              // Field containing the URI string. Required.
	IgnoreMissing      *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`             // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	KeepOriginal       *bool  `json:"keep_original,omitempty" jsonschema:"default=true" yaml:"keep_original,omitempty"`                // If `true`, the processor copies the unparsed URI to `<target_field>.original`.
	RemoveIfSuccessful *bool  `json:"remove_if_successful,omitempty" jsonschema:"default=false" yaml:"remove_if_successful,omitempty"` // If `true`, the processor removes the `field` after parsing the URI string. If parsing fails, the processor does not remove the `field`.
	TargetField        *Field `json:"target_field,omitempty" jsonschema:"default=url" yaml:"target_field,omitempty"`                   // Output field for the URI object.
}

Parses a Uniform Resource Identifier (URI) string and extracts its components as an object. This URI object includes properties for the URI’s domain, path, fragment, port, query, scheme, user info, username, and password.

type UrlDecodeProcessor

type UrlDecodeProcessor struct {
	ProcessorBase
	Field         Field  `json:"field" yaml:"field"`                                                                  // The field to decode. Required.
	IgnoreMissing *bool  `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"` // If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document.
	TargetField   *Field `json:"target_field,omitempty" jsonschema:"default=field" yaml:"target_field,omitempty"`     // The field to assign the converted value to. By default, the field is updated in-place.
}

URL-decodes a string. If the field is an array of strings, all members of the array will be decoded.

Reference: urldecode processor

type UserAgentProcessor

type UserAgentProcessor struct {
	ProcessorBase
	Field         Field               `json:"field" yaml:"field"`                                                                   // The field containing the user agent string. Required.
	IgnoreMissing *bool               `json:"ignore_missing,omitempty" jsonschema:"default=false" yaml:"ignore_missing,omitempty"`  // If `true` and `field` does not exist, the processor quietly exits without modifying the document.
	RegexFile     *string             `json:"regex_file,omitempty" yaml:"regex_file,omitempty"`                                     // The name of the file in the `config/ingest-user-agent` directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the `regexes.yaml` from uap-core it ships with.
	TargetField   *Field              `json:"target_field,omitempty" jsonschema:"default=user_agent" yaml:"target_field,omitempty"` // The field that will be filled with the user agent details.
	Properties    []UserAgentProperty ``                                                                                            // Controls what properties are added to `target_field`.
	/* 213-byte string literal not displayed */
	ExtractDeviceType *bool `json:"extract_device_type,omitempty" jsonschema:"default=false" yaml:"extract_device_type,omitempty"` // Extracts device type from the user agent string on a best-effort basis.
}

The `user_agent` processor extracts details from the user agent string a browser sends with its web requests. This processor adds this information by default under the `user_agent` field.

Reference: user_agent processor

type UserAgentProperty

type UserAgentProperty string

const (
	NameUserAgentProperty     UserAgentProperty = "name"
	OSUserAgentProperty       UserAgentProperty = "os"
	DeviceUserAgentProperty   UserAgentProperty = "device"
	OriginalUserAgentProperty UserAgentProperty = "original"
	VersionUserAgentProperty  UserAgentProperty = "version"
)

type VersionNumber

type VersionNumber int64