log

package
v0.0.0-...-e951c9a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 6, 2022 License: AGPL-3.0 Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	IPv4Charset = "0123456789."
	IPv6Charset = "0123456789abcdefABCDEF:."
)
View Source 
const (
	ConvertBytes    = "bytes"
	ConvertDuration = "duration"
	ConvertFloat    = "float"
)
View Source 
const MaxInternedStrings = 1024

Variables

View Source 
var (
	ErrIPFilterInvalidPattern   = errors.New("ip: invalid pattern")
	ErrIPFilterInvalidOperation = errors.New("ip: invalid operation")
)
View Source 
var EmptyLabelsResult = NewLabelsResult(labels.Labels{}, labels.Labels{}.Hash())
View Source 
var (

	// NoopLabelFilter is a label filter that doesn't filter out any values.
	NoopLabelFilter = noopLabelFilter{}
)
View Source 
var TrueFilter = trueFilter{}

TrueFilter is a filter that returns and matches all log lines whatever their content.

Functions

func IsNoopPipeline 

func IsNoopPipeline(p Pipeline) bool

IsNoopPipeline tells if a pipeline is a Noop.

Types

type AnalyzablePipeline 

type AnalyzablePipeline interface {
	Pipeline
	Stages() []Stage
	LabelsBuilder() *BaseLabelsBuilder
}

type BaseLabelsBuilder 

type BaseLabelsBuilder struct {
	// contains filtered or unexported fields
}

BaseLabelsBuilder is a label builder used by pipeline and stages. Only one base builder is used and it contains cache for each LabelsBuilders.

func NewBaseLabelsBuilder 

func NewBaseLabelsBuilder() *BaseLabelsBuilder

NewLabelsBuilder creates a new base labels builder.

func NewBaseLabelsBuilderWithGrouping 

func NewBaseLabelsBuilderWithGrouping(groups []string, parserKeyHints ParserHint, without, noLabels bool) *BaseLabelsBuilder

NewBaseLabelsBuilderWithGrouping creates a new base labels builder with grouping to compute results.

func (*BaseLabelsBuilder) ForLabels 

func (b *BaseLabelsBuilder) ForLabels(lbs labels.Labels, hash uint64) *LabelsBuilder

ForLabels creates a labels builder for a given labels set as base. The labels cache is shared across all created LabelsBuilders.

func (BaseLabelsBuilder) Hash 

func (h BaseLabelsBuilder) Hash(lbs labels.Labels) uint64

Hash hashes the labels

func (*BaseLabelsBuilder) ParserLabelHints 

func (b *BaseLabelsBuilder) ParserLabelHints() ParserHint

ParserLabelHints returns a limited list of expected labels to extract for metric queries. Returns nil when it's impossible to hint labels extractions.

type BinaryLabelFilter 

type BinaryLabelFilter struct {
	Left  LabelFilterer
	Right LabelFilterer
	// contains filtered or unexported fields
}

func NewAndLabelFilter 

func NewAndLabelFilter(left LabelFilterer, right LabelFilterer) *BinaryLabelFilter

NewAndLabelFilter creates a new LabelFilterer from a and binary operation of two LabelFilterer.

func NewOrLabelFilter 

func NewOrLabelFilter(left LabelFilterer, right LabelFilterer) *BinaryLabelFilter

NewOrLabelFilter creates a new LabelFilterer from a or binary operation of two LabelFilterer.

func (*BinaryLabelFilter) Process 

func (b *BinaryLabelFilter) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*BinaryLabelFilter) RequiredLabelNames 

func (b *BinaryLabelFilter) RequiredLabelNames() []string

func (*BinaryLabelFilter) String 

func (b *BinaryLabelFilter) String() string

type BytesLabelFilter 

type BytesLabelFilter struct {
	Name  string
	Value uint64
	Type  LabelFilterType
}

func NewBytesLabelFilter 

func NewBytesLabelFilter(t LabelFilterType, name string, b uint64) *BytesLabelFilter

NewBytesLabelFilter creates a new label filterer which parses bytes string representation (1KB) from the value of the named label and compares it with the given b value.

func (*BytesLabelFilter) Process 

func (d *BytesLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*BytesLabelFilter) RequiredLabelNames 

func (d *BytesLabelFilter) RequiredLabelNames() []string

func (*BytesLabelFilter) String 

func (d *BytesLabelFilter) String() string

type DurationLabelFilter 

type DurationLabelFilter struct {
	Name  string
	Value time.Duration
	Type  LabelFilterType
}

func NewDurationLabelFilter 

func NewDurationLabelFilter(t LabelFilterType, name string, d time.Duration) *DurationLabelFilter

NewDurationLabelFilter creates a new label filterer which parses duration string representation (5s) from the value of the named label and compares it with the given d value.

func (*DurationLabelFilter) Process 

func (d *DurationLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*DurationLabelFilter) RequiredLabelNames 

func (d *DurationLabelFilter) RequiredLabelNames() []string

func (*DurationLabelFilter) String 

func (d *DurationLabelFilter) String() string

type Filterer 

type Filterer interface {
	Filter(line []byte) bool
	ToStage() Stage
}

Filterer is a interface to filter log lines.

func NewAndFilter 

func NewAndFilter(left Filterer, right Filterer) Filterer

NewAndFilter creates a new filter which matches only if left and right matches.

func NewAndFilters 

func NewAndFilters(filters []Filterer) Filterer

NewAndFilters creates a new filter which matches only if all filters match

func NewFilter 

func NewFilter(match string, mt labels.MatchType) (Filterer, error)

NewFilter creates a new line filter from a match string and type.

type FiltererFunc 

type FiltererFunc func(line []byte) bool

LineFilterFunc is a syntax sugar for creating line filter from a function

func (FiltererFunc) Filter 

func (f FiltererFunc) Filter(line []byte) bool

type IPLabelFilter 

type IPLabelFilter struct {
	// contains filtered or unexported fields
}

func NewIPLabelFilter 

func NewIPLabelFilter(pattern string, label string, ty LabelFilterType) *IPLabelFilter

NewIPLabelFilter is used to construct ip filter as label filter for the given `label`.

func (*IPLabelFilter) PatternError 

func (f *IPLabelFilter) PatternError() error

PatternError will be used `labelFilter.Stage()` method so that, if the given pattern is wrong it returns proper 400 error to the client of LogQL.

func (*IPLabelFilter) Process 

func (f *IPLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

`Process` implements `Stage` interface

func (*IPLabelFilter) RequiredLabelNames 

func (f *IPLabelFilter) RequiredLabelNames() []string

`RequiredLabelNames` implements `Stage` interface

func (*IPLabelFilter) String 

func (f *IPLabelFilter) String() string

`String` implements fmt.Stringer inteface, by which also implements `LabelFilterer` inteface.

type IPLineFilter 

type IPLineFilter struct {
	// contains filtered or unexported fields
}

func NewIPLineFilter 

func NewIPLineFilter(pattern string, ty labels.MatchType) (*IPLineFilter, error)

NewIPLineFilter is used to construct ip filter as a `LineFilter`

func (*IPLineFilter) Filter 

func (f *IPLineFilter) Filter(line []byte) bool

Filter implement `Filterer` interface. Used by `LineFilter`

func (*IPLineFilter) Process 

func (f *IPLineFilter) Process(_ int64, line []byte, _ *LabelsBuilder) ([]byte, bool)

`Process` implements `Stage` interface

func (*IPLineFilter) RequiredLabelNames 

func (f *IPLineFilter) RequiredLabelNames() []string

`RequiredLabelNames` implements `Stage` interface

func (*IPLineFilter) ToStage 

func (f *IPLineFilter) ToStage() Stage

ToStage implements `Filterer` interface.

type IPMatchType 

type IPMatchType int

type IPMatcher 

type IPMatcher interface{}

Should be one of the netaddr.IP, netaddr.IPRange, netadd.IPPrefix.

type JSONExpression 

type JSONExpression struct {
	Identifier string
	Expression string
}

func NewJSONExpr 

func NewJSONExpr(identifier, expression string) JSONExpression

type JSONExpressionParser 

type JSONExpressionParser struct {
	// contains filtered or unexported fields
}

func NewJSONExpressionParser 

func NewJSONExpressionParser(expressions []JSONExpression) (*JSONExpressionParser, error)

func (*JSONExpressionParser) Process 

func (j *JSONExpressionParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*JSONExpressionParser) RequiredLabelNames 

func (j *JSONExpressionParser) RequiredLabelNames() []string

type JSONParser 

type JSONParser struct {
	// contains filtered or unexported fields
}

func NewJSONParser 

func NewJSONParser() *JSONParser

NewJSONParser creates a log stage that can parse a json log line and add properties as labels.

func (*JSONParser) Process 

func (j *JSONParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*JSONParser) RequiredLabelNames 

func (j *JSONParser) RequiredLabelNames() []string

type LabelFilterType 

type LabelFilterType int

LabelFilterType is an enum for label filtering types. 

const (
	LabelFilterEqual LabelFilterType = iota
	LabelFilterNotEqual
	LabelFilterGreaterThan
	LabelFilterGreaterThanOrEqual
	LabelFilterLesserThan
	LabelFilterLesserThanOrEqual
)

Possible LabelFilterType.

func (LabelFilterType) String 

func (f LabelFilterType) String() string

type LabelFilterer 

type LabelFilterer interface {
	Stage
	fmt.Stringer
}

LabelFilterer can filter extracted labels.

func ReduceAndLabelFilter 

func ReduceAndLabelFilter(filters []LabelFilterer) LabelFilterer

ReduceAndLabelFilter Reduces multiple label filterer into one using binary and operation.

type LabelFmt 

type LabelFmt struct {
	Name  string
	Value string

	Rename bool
}

LabelFmt is a configuration struct for formatting a label.

func NewRenameLabelFmt 

func NewRenameLabelFmt(dst, target string) LabelFmt

NewRenameLabelFmt creates a configuration to rename a label.

func NewTemplateLabelFmt 

func NewTemplateLabelFmt(dst, template string) LabelFmt

NewTemplateLabelFmt creates a configuration to format a label using text template.

type LabelsBuilder 

type LabelsBuilder struct {
	*BaseLabelsBuilder
	// contains filtered or unexported fields
}

LabelsBuilder is the same as labels.Builder but tailored for this package.

func (*LabelsBuilder) BaseHas 

func (b *LabelsBuilder) BaseHas(key string) bool

BaseHas returns the base labels have the given key

func (*LabelsBuilder) Del 

func (b *LabelsBuilder) Del(ns ...string) *LabelsBuilder

Del deletes the label of the given name.

func (*LabelsBuilder) Get 

func (b *LabelsBuilder) Get(key string) (string, bool)

Get returns the value of a labels key if it exists.

func (*LabelsBuilder) GetErr 

func (b *LabelsBuilder) GetErr() string

GetErr return the current error label value.

func (*LabelsBuilder) GetErrorDetails 

func (b *LabelsBuilder) GetErrorDetails() string

func (*LabelsBuilder) GroupedLabels 

func (b *LabelsBuilder) GroupedLabels() LabelsResult

GroupedLabels returns the LabelsResult from the builder. Groups are applied and the cache is used when possible.

func (*LabelsBuilder) HasErr 

func (b *LabelsBuilder) HasErr() bool

HasErr tells if the error label has been set.

func (*LabelsBuilder) HasErrorDetails 

func (b *LabelsBuilder) HasErrorDetails() bool

func (LabelsBuilder) Hash 

func (h LabelsBuilder) Hash(lbs labels.Labels) uint64

Hash hashes the labels

func (*LabelsBuilder) LabelsResult 

func (b *LabelsBuilder) LabelsResult() LabelsResult

LabelsResult returns the LabelsResult from the builder. No grouping is applied and the cache is used when possible.

func (*LabelsBuilder) Map 

func (b *LabelsBuilder) Map() map[string]string

func (*LabelsBuilder) Reset 

func (b *LabelsBuilder) Reset()

Reset clears all current state for the builder.

func (*LabelsBuilder) Set 

func (b *LabelsBuilder) Set(n, v string) *LabelsBuilder

Set the name/value pair as a label.

func (*LabelsBuilder) SetErr 

func (b *LabelsBuilder) SetErr(err string) *LabelsBuilder

SetErr sets the error label.

func (*LabelsBuilder) SetErrorDetails 

func (b *LabelsBuilder) SetErrorDetails(desc string) *LabelsBuilder

func (*LabelsBuilder) UnsortedLabels 

func (b *LabelsBuilder) UnsortedLabels(buf labels.Labels) labels.Labels

type LabelsFormatter 

type LabelsFormatter struct {
	// contains filtered or unexported fields
}

func NewLabelsFormatter 

func NewLabelsFormatter(fmts []LabelFmt) (*LabelsFormatter, error)

NewLabelsFormatter creates a new formatter that can format multiple labels at once. Either by renaming or using text template. It is not allowed to reformat the same label twice within the same formatter.

func (*LabelsFormatter) Process 

func (lf *LabelsFormatter) Process(ts int64, l []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LabelsFormatter) RequiredLabelNames 

func (lf *LabelsFormatter) RequiredLabelNames() []string

type LabelsResult 

type LabelsResult interface {
	String() string
	Labels() labels.Labels
	Hash() uint64
}

LabelsResult is a computed labels result that contains the labels set with associated string and hash. The is mainly used for caching and returning labels computations out of pipelines and stages.

func NewLabelsResult 

func NewLabelsResult(lbs labels.Labels, hash uint64) LabelsResult

NewLabelsResult creates a new LabelsResult from a labels set and a hash.

type LineExtractor 

type LineExtractor func([]byte) float64

LineExtractor extracts a float64 from a log line. 

var (
	CountExtractor LineExtractor = func(line []byte) float64 { return 1. }
	BytesExtractor LineExtractor = func(line []byte) float64 { return float64(len(line)) }
)

type LineFormatter 

type LineFormatter struct {
	*template.Template
	// contains filtered or unexported fields
}

func NewFormatter 

func NewFormatter(tmpl string) (*LineFormatter, error)

NewFormatter creates a new log line formatter from a given text template.

func (*LineFormatter) Process 

func (lf *LineFormatter) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LineFormatter) RequiredLabelNames 

func (lf *LineFormatter) RequiredLabelNames() []string

type LogfmtParser 

type LogfmtParser struct {
	// contains filtered or unexported fields
}

func NewLogfmtParser 

func NewLogfmtParser() *LogfmtParser

NewLogfmtParser creates a parser that can extract labels from a logfmt log line. Each keyval is extracted into a respective label.

func (*LogfmtParser) Process 

func (l *LogfmtParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LogfmtParser) RequiredLabelNames 

func (l *LogfmtParser) RequiredLabelNames() []string

type NumericLabelFilter 

type NumericLabelFilter struct {
	Name  string
	Value float64
	Type  LabelFilterType
}

func NewNumericLabelFilter 

func NewNumericLabelFilter(t LabelFilterType, name string, v float64) *NumericLabelFilter

NewNumericLabelFilter creates a new label filterer which parses float64 string representation (5.2) from the value of the named label and compares it with the given f value.

func (*NumericLabelFilter) Process 

func (n *NumericLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*NumericLabelFilter) RequiredLabelNames 

func (n *NumericLabelFilter) RequiredLabelNames() []string

func (*NumericLabelFilter) String 

func (n *NumericLabelFilter) String() string

type ParserHint 

type ParserHint interface {
	// Tells if a label with the given key should be extracted.
	ShouldExtract(key string) bool
	// Tells if there's any hint that start with the given prefix.
	// This allows to speed up key searching in nested structured like json.
	ShouldExtractPrefix(prefix string) bool
	// Tells if we should not extract any labels.
	// For example in :
	//		 sum(rate({app="foo"} | json [5m]))
	// We don't need to extract any labels from the log line.
	NoLabels() bool
}

ParserHint are hints given to LogQL parsers. This is specially useful for parser that extract implicitly all possible label keys. This is used only within metric queries since it's rare that you need all label keys. For example in the following expression: 

sum by (status_code) (rate({app="foo"} | json [5m]))

All we need to extract is the status_code in the json parser.

type PatternParser 

type PatternParser struct {
	// contains filtered or unexported fields
}

func NewPatternParser 

func NewPatternParser(pn string) (*PatternParser, error)

func (*PatternParser) Process 

func (l *PatternParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*PatternParser) RequiredLabelNames 

func (l *PatternParser) RequiredLabelNames() []string

type Pipeline 

type Pipeline interface {
	ForStream(labels labels.Labels) StreamPipeline
}

Pipeline can create pipelines for each log stream.

func NewFilteringPipeline 

func NewFilteringPipeline(f []PipelineFilter, p Pipeline) Pipeline

NewFilteringPipeline creates a pipeline where entries from the underlying log stream are filtered by pipeline filters before being passed to the pipeline representing the queried data. Filters are always upstream of the pipeline

func NewNoopPipeline 

func NewNoopPipeline() Pipeline

NewNoopPipeline creates a pipelines that does not process anything and returns log streams as is.

func NewPipeline 

func NewPipeline(stages []Stage) Pipeline

NewPipeline creates a new pipeline for a given set of stages.

type PipelineFilter 

type PipelineFilter struct {
	Start    int64
	End      int64
	Matchers []*labels.Matcher
	Pipeline Pipeline
}

PipelineFilter contains a set of matchers and a pipeline that, when matched, causes an entry from a log stream to be skipped. Matching entries must also fall between 'start' and 'end', inclusive

type RegexpParser 

type RegexpParser struct {
	// contains filtered or unexported fields
}

func NewRegexpParser 

func NewRegexpParser(re string) (*RegexpParser, error)

NewRegexpParser creates a new log stage that can extract labels from a log line using a regex expression. The regex expression must contains at least one named match. If the regex doesn't match the line is not filtered out.

func (*RegexpParser) Process 

func (r *RegexpParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*RegexpParser) RequiredLabelNames 

func (r *RegexpParser) RequiredLabelNames() []string

type SampleExtractor 

type SampleExtractor interface {
	ForStream(labels labels.Labels) StreamSampleExtractor
}

SampleExtractor creates StreamSampleExtractor that can extract samples for a given log stream.

func LabelExtractorWithStages 

func LabelExtractorWithStages(
	labelName, conversion string,
	groups []string, without, noLabels bool,
	preStages []Stage,
	postFilter Stage,
) (SampleExtractor, error)

LabelExtractorWithStages creates a SampleExtractor that will extract metrics from a labels. A set of log stage is executed before the conversion. A Filtering stage is executed after the conversion allowing to remove sample containing the __error__ label.

func NewFilteringSampleExtractor 

func NewFilteringSampleExtractor(f []PipelineFilter, e SampleExtractor) SampleExtractor

NewFilteringSampleExtractor creates a sample extractor where entries from the underlying log stream are filtered by pipeline filters before being passed to extract samples. Filters are always upstream of the extractor.

func NewLineSampleExtractor 

func NewLineSampleExtractor(ex LineExtractor, stages []Stage, groups []string, without, noLabels bool) (SampleExtractor, error)

NewLineSampleExtractor creates a SampleExtractor from a LineExtractor. Multiple log stages are run before converting the log line.

type Stage 

type Stage interface {
	Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)
	RequiredLabelNames() []string
}

Stage is a single step of a Pipeline. A Stage implementation should never mutate the line passed, but instead either return the line unchanged or allocate a new line. 

var NoopStage Stage = &noopStage{}

NoopStage is a stage that doesn't process a log line.

func ReduceStages 

func ReduceStages(stages []Stage) Stage

ReduceStages reduces multiple stages into one.

type StageFunc 

type StageFunc struct {
	// contains filtered or unexported fields
}

func (StageFunc) Process 

func (fn StageFunc) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (StageFunc) RequiredLabelNames 

func (fn StageFunc) RequiredLabelNames() []string

type StreamPipeline 

type StreamPipeline interface {
	BaseLabels() LabelsResult
	// Process processes a log line and returns the transformed line and the labels.
	// The buffer returned for the log line can be reused on subsequent calls to Process and therefore must be copied.
	Process(ts int64, line []byte) (resultLine []byte, resultLabels LabelsResult, matches bool)
	ProcessString(ts int64, line string) (resultLine string, resultLabels LabelsResult, matches bool)
}

StreamPipeline transform and filter log lines and labels. A StreamPipeline never mutate the received line.

func NewStreamPipeline 

func NewStreamPipeline(stages []Stage, labelsBuilder *LabelsBuilder) StreamPipeline

type StreamSampleExtractor 

type StreamSampleExtractor interface {
	BaseLabels() LabelsResult
	Process(ts int64, line []byte) (float64, LabelsResult, bool)
	ProcessString(ts int64, line string) (float64, LabelsResult, bool)
}

StreamSampleExtractor extracts sample for a log line. A StreamSampleExtractor never mutate the received line.

type StringLabelFilter 

type StringLabelFilter struct {
	*labels.Matcher
}

func NewStringLabelFilter 

func NewStringLabelFilter(m *labels.Matcher) *StringLabelFilter

NewStringLabelFilter creates a new label filterer which compares string label. This is the only LabelFilterer that can filter out the __error__ label. Unlike other LabelFilterer which apply conversion, if the label name doesn't exist it is compared with an empty value.

func (*StringLabelFilter) Process 

func (s *StringLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*StringLabelFilter) RequiredLabelNames 

func (s *StringLabelFilter) RequiredLabelNames() []string

type UnpackParser 

type UnpackParser struct {
	// contains filtered or unexported fields
}

func NewUnpackParser 

func NewUnpackParser() *UnpackParser

NewUnpackParser creates a new unpack stage. The unpack stage will parse a json log line as map[string]string where each key will be translated into labels. A special key _entry will also be used to replace the original log line. This is to be used in conjunction with Promtail pack stage. see https://grafana.com/docs/loki/latest/clients/promtail/stages/pack/

func (*UnpackParser) Process 

func (u *UnpackParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (UnpackParser) RequiredLabelNames 

func (UnpackParser) RequiredLabelNames() []string

Source Files

View all Source files

Directories

Path Synopsis
Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.
 Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.