ingest

Documentation

Overview

Package ingest provides the datatypes and functions helping with ingesting data into Axiom.

Usage:

import "github.com/axiomhq/axiom-go/axiom/ingest"

Index

• Constants
• type Failure
• type Option
  • func SetCSVDelimiter(delim string) Option
  • func SetEventLabel(key string, value any) Option
  • func SetEventLabels(labels map[string]any) Option
  • func SetTimestampField(field string) Option
  • func SetTimestampFormat(format string) Option
• type Options
• type Status
  • func (s *Status) Add(other *Status)

Constants

const TimestampField = "_time"

TimestampField is the default field the server will look for a timestamp to use as the ingestion time. If not present, the server will set the ingestion time to the current server time.

Types

type Failure

type Failure struct {
	// Timestamp of the event that failed to ingest.
	Timestamp time.Time `json:"timestamp"`
	// Error that made the event fail to ingest.
	Error string `json:"error"`
}

Failure describes the ingestion failure of a single event.

type Option

type Option func(*Options)

An Option applies optional parameters to an ingest operation.

func SetCSVDelimiter

func SetCSVDelimiter(delim string) Option

SetCSVDelimiter specifies the delimiter that separates CSV fields. Only valid when the content to be ingested is CSV formatted.

func SetEventLabel

func SetEventLabel(key string, value any) Option

SetEventLabel adds a label to apply to all events. This option can be called multiple times to add multiple labels. If a label with the same key already exists, it will be overwritten.

func SetEventLabels

func SetEventLabels(labels map[string]any) Option

SetEventLabels sets the labels to apply to all events. It will overwrite any existing labels.

func SetTimestampField

func SetTimestampField(field string) Option

SetTimestampField specifies the field Axiom will use to extract the events time from. Defaults to TimestampField

func SetTimestampFormat

func SetTimestampFormat(format string) Option

SetTimestampFormat specifies the format of the timestamp field. The reference time is "Mon Jan 2 15:04:05 -0700 MST 2006", as specified in https://pkg.go.dev/time/?tab=doc#Parse.

type Options

type Options struct {
	// TimestampField defines a custom field to extract the ingestion timestamp
	// from. Defaults to [TimestampField].
	TimestampField string `url:"timestamp-field,omitempty"`
	// TimestampFormat defines a custom format for the [Options.TimestampField].
	// The reference time is "Mon Jan 2 15:04:05 -0700 MST 2006", as specified
	// in https://pkg.go.dev/time/?tab=doc#Parse.
	TimestampFormat string `url:"timestamp-format,omitempty"`
	// CSVDelimiter is the delimiter that separates CSV fields. Only valid when
	// the content to be ingested is CSV formatted.
	CSVDelimiter string `url:"csv-delimiter,omitempty"`
	// EventLabels are a key-value pairs that will be added to all events. Their
	// purpose is to allow for labeling events without alterting the original
	// event data. This is especially useful when ingesting events from a
	// third-party source that you do not have control over.
	EventLabels map[string]any `url:"-"`
}

Options specifies the optional parameters for ingestion.

type Status

type Status struct {
	// Ingested is the amount of events that have been ingested.
	Ingested uint64 `json:"ingested"`
	// Failed is the amount of events that failed to ingest.
	Failed uint64 `json:"failed"`
	// Failures are the ingestion failures, if any.
	Failures []*Failure `json:"failures"`
	// ProcessedBytes is the number of bytes processed.
	ProcessedBytes uint64 `json:"processedBytes"`
	// BlocksCreated is the amount of blocks created.
	//
	// Deprecated: BlocksCreated is deprecated and will be removed in a future
	// release.
	BlocksCreated uint32 `json:"blocksCreated"`
	// WALLength is the length of the Write-Ahead Log.
	//
	// Deprecated: WALLength is deprecated and will be removed in a future
	// release.
	WALLength uint32 `json:"walLength"`
	// TraceID is the ID of the trace that was generated by the server for this
	// statuses ingest request.
	TraceID string `json:"-"`
}

Status is the status of an event ingestion operation.

func (*Status) Add

func (s *Status) Add(other *Status)

Add adds the status of another ingestion operation to the current status. The trace ID is ignored.