Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // MaxLineSizeBytes is the maximum length of a single line for /insert/* handlers MaxLineSizeBytes = flagutil.NewBytes("insert.maxLineSizeBytes", 256*1024, "The maximum size of a single line, which can be read by /insert/* handlers; "+ "see https://docs.victoriametrics.com/victorialogs/faq/#what-length-a-log-record-is-expected-to-have") // MaxFieldsPerLine is the maximum number of fields per line for /insert/* handlers MaxFieldsPerLine = flag.Int("insert.maxFieldsPerLine", 1000, "The maximum number of log fields per line, which can be read by /insert/* handlers; "+ "see https://docs.victoriametrics.com/victorialogs/faq/#how-many-fields-a-single-log-entry-may-contain") )
Functions ¶
func ExtractTimestampFromFields ¶
func ExtractTimestampFromFields(timeField string, fields []logstorage.Field) (int64, error)
ExtractTimestampFromFields extracts timestamp in nanoseconds from the field with the name timeField at fields.
The value for the timeField is set to empty string after returning from the function, so it could be ignored during data ingestion.
The current timestamp is returned if fields do not contain a field with timeField name or if the timeField value is empty.
func ParseUnixTimestamp ¶
ParseUnixTimestamp parses s as unix timestamp in seconds, milliseconds, microseconds or nanoseconds and returns the parsed timestamp in nanoseconds.
Types ¶
type BenchmarkLogMessageProcessor ¶
type BenchmarkLogMessageProcessor struct{}
BenchmarkLogMessageProcessor implements LogMessageProcessor for benchmarks.
func (*BenchmarkLogMessageProcessor) AddRow ¶
func (blp *BenchmarkLogMessageProcessor) AddRow(_ int64, _, _ []logstorage.Field)
AddRow implements LogMessageProcessor interface.
func (*BenchmarkLogMessageProcessor) MustClose ¶
func (blp *BenchmarkLogMessageProcessor) MustClose()
MustClose implements LogMessageProcessor interface.
type CommonParams ¶
type CommonParams struct {
TenantID logstorage.TenantID
TimeField string
MsgFields []string
StreamFields []string
IgnoreFields []string
ExtraFields []logstorage.Field
Debug bool
DebugRequestURI string
DebugRemoteAddr string
}
CommonParams contains common HTTP parameters used by log ingestion APIs.
See https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters
func GetCommonParams ¶
func GetCommonParams(r *http.Request) (*CommonParams, error)
GetCommonParams returns CommonParams from r.
func GetCommonParamsForSyslog ¶
func GetCommonParamsForSyslog(tenantID logstorage.TenantID, streamFields, ignoreFields []string, extraFields []logstorage.Field) *CommonParams
GetCommonParamsForSyslog returns common params needed for parsing syslog messages and storing them to the given tenantID.
func (*CommonParams) NewLogMessageProcessor ¶
func (cp *CommonParams) NewLogMessageProcessor(protocolName string, isStreamMode bool) LogMessageProcessor
NewLogMessageProcessor returns new LogMessageProcessor for the given cp.
MustClose() must be called on the returned LogMessageProcessor when it is no longer needed.
type InsertRowProcessor ¶ added in v1.102.19
type InsertRowProcessor interface {
// AddInsertRow must add r to the underlying storage.
AddInsertRow(r *logstorage.InsertRow)
}
InsertRowProcessor is used by native data ingestion protocol parser.
type LineReader ¶
type LineReader struct {
// Line contains the next line read after the call to NextLine
//
// The Line contents is valid until the next call to NextLine.
Line []byte
// contains filtered or unexported fields
}
LineReader reads newline-delimited lines from the underlying reader
func NewLineReader ¶
func NewLineReader(name string, r io.Reader) *LineReader
NewLineReader returns LineReader for r.
func (*LineReader) Err ¶
func (lr *LineReader) Err() error
Err returns the last error after NextLine call.
func (*LineReader) NextLine ¶
func (lr *LineReader) NextLine() bool
NextLine reads the next line from the underlying reader.
It returns true if the next line is successfully read into Line. If the line length exceeds MaxLineSizeBytes, then this line is skipped and an empty line is returned instead.
If false is returned, then no more lines left to read from r. Check for Err in this case.
type LogMessageProcessor ¶
type LogMessageProcessor interface {
// AddRow must add row to the LogMessageProcessor with the given timestamp and fields.
//
// If streamFields is non-nil, then the given streamFields must be used as log stream fields instead of pre-configured fields.
//
// The LogMessageProcessor implementation cannot hold references to fields, since the caller can reuse them.
AddRow(timestamp int64, fields, streamFields []logstorage.Field)
// MustClose() must flush all the remaining fields and free up resources occupied by LogMessageProcessor.
MustClose()
}
LogMessageProcessor is an interface for log message processors.
type TestLogMessageProcessor ¶
type TestLogMessageProcessor struct {
// contains filtered or unexported fields
}
TestLogMessageProcessor implements LogMessageProcessor for testing.
func (*TestLogMessageProcessor) AddRow ¶
func (tlp *TestLogMessageProcessor) AddRow(timestamp int64, fields, streamFields []logstorage.Field)
AddRow adds row with the given timestamp and fields to tlp
func (*TestLogMessageProcessor) MustClose ¶
func (tlp *TestLogMessageProcessor) MustClose()
MustClose closes tlp.
Source Files