slowlog

README

MySQL Slow Log Parsing and Aggregation

This package provides functions for working with the MySQL slow log.

Acknowledgement

This code was originally copied from percona/go-mysql @ 2a6037d7d809b18ebd6d735b397f2321879af611. See that project for original contributors and copyright.

This project is a fork to continue development of percona/go-mysql as separate packages. GitHub only allows forking a project once, which is why the code has been copied.

Documentation

Overview

Package slowlog provides functions and data structures for working with the MySQL slow log.

Index

• Constants
• Variables
• type Aggregator
  • func NewAggregator(samples bool, utcOffset time.Duration, outlierTime float64) *Aggregator
  • func (a *Aggregator) AddEvent(event Event, id, fingerprint string)
  • func (a *Aggregator) Finalize() Result
• type BoolStats
• type Class
  • func NewAggregateClass(id, fingerprint string, members []*Class) *Class
  • func NewClass(id, fingerprint string, sample bool) *Class
  • func (c *Class) AddEvent(e Event, outlier bool)
  • func (c *Class) Finalize(rateLimit uint)
• type Event
  • func NewEvent() *Event
• type Example
• type FileParser
  • func NewFileParser(file *os.File) *FileParser
  • func (p *FileParser) Error() error
  • func (p *FileParser) Events() <-chan Event
  • func (p *FileParser) Start(opt Options) error
  • func (p *FileParser) Stop()
• type Metrics
  • func NewMetrics() Metrics
  • func (m *Metrics) AddEvent(e Event, outlier bool)
  • func (m *Metrics) Finalize(rateLimit uint)
• type NumberStats
• type Options
• type Parser
• type Result
• type TimeStats

Constants

const (
	// MAX_EXAMPLE_BYTES defines the maximum Example.Query size.
	MAX_EXAMPLE_BYTES = 1024 * 10
)

Variables

var Debug = false

var (
	// ErrStarted is returned if Parser.Start is called more than once.
	ErrStarted = errors.New("parser is started")
)

Types

type Aggregator

type Aggregator struct {
	// contains filtered or unexported fields
}

An Aggregator groups events by class ID. When there are no more events, a call to Finalize computes all metric statistics and returns a Result.

func NewAggregator

func NewAggregator(samples bool, utcOffset time.Duration, outlierTime float64) *Aggregator

NewAggregator returns a new Aggregator.

func (*Aggregator) AddEvent

func (a *Aggregator) AddEvent(event Event, id, fingerprint string)

AddEvent adds the event to the aggregator, automatically creating new classes as needed.

func (*Aggregator) Finalize

func (a *Aggregator) Finalize() Result

Finalize calculates all metric statistics and returns a Result. Call this function when done adding events to the aggregator.

type BoolStats

type BoolStats struct {
	Sum uint64 // %true = Sum/Cnt
	// contains filtered or unexported fields
}

BoolStats are boolean-based metrics like QC_Hit and Filesort.

type Class

type Class struct {
	Id            string   // 32-character hex checksum of fingerprint
	Fingerprint   string   // canonical form of query: values replaced with "?"
	Metrics       Metrics  // statistics for each metric, e.g. max Query_time
	TotalQueries  uint64   // total number of queries in class
	UniqueQueries uint     // unique number of queries in class
	Example       *Example `json:",omitempty"` // sample query with max Query_time
	// contains filtered or unexported fields
}

A Class represents all events with the same fingerprint and class ID. This is only enforced by convention, so be careful not to mix events from different classes.

func NewAggregateClass

func NewAggregateClass(id, fingerprint string, members []*Class) *Class

NewAggregateClass makes a new Class from the given member classes.

func NewClass

func NewClass(id, fingerprint string, sample bool) *Class

NewClass returns a new Class for the class ID and fingerprint. If sample is true, the query with the greatest Query_time is saved.

func (*Class) AddEvent

func (c *Class) AddEvent(e Event, outlier bool)

AddEvent adds an event to the query class.

func (*Class) Finalize

func (c *Class) Finalize(rateLimit uint)

Finalize calculates all metric statistics. Call this function when done adding events to the class.

type Event

type Event struct {
	Offset        uint64 // byte offset in file at which event starts
	Ts            string // raw timestamp of event
	Admin         bool   // true if Query is admin command
	Query         string // SQL query or admin command
	User          string
	Host          string
	Db            string
	TimeMetrics   map[string]float64 // *_time and *_wait metrics
	NumberMetrics map[string]uint64  // most metrics
	BoolMetrics   map[string]bool    // yes/no metrics
	RateType      string             // Percona Server rate limit type
	RateLimit     uint               // Percona Server rate limit value
}

An Event is a query like "SELECT col FROM t WHERE id = 1", some metrics like Query_time (slow log) or SUM_TIMER_WAIT (Performance Schema), and other metadata like default database, timestamp, etc. Metrics and metadata are not guaranteed to be defined--and frequently they are not--but at minimum an event is expected to define the query and Query_time metric. Other metrics and metadata vary according to MySQL version, distro, and configuration.

func NewEvent

func NewEvent() *Event

NewEvent returns a new Event with initialized metric maps.

type Example

type Example struct {
	QueryTime float64 // Query_time
	Db        string  // Schema: <db> or USE <db>
	Query     string  // truncated to MAX_EXAMPLE_BYTES
	Ts        string  `json:",omitempty"` // in MySQL time zone
}

A Example is a real query and its database, timestamp, and Query_time. If the query is larger than MAX_EXAMPLE_BYTES, it is truncated and "..." is appended.

type FileParser

type FileParser struct {
	*sync.Mutex
	// contains filtered or unexported fields
}

FileParser represents a file-based Parser. This is the canonical Parser because the slow log is a file.

func NewFileParser

func NewFileParser(file *os.File) *FileParser

NewFileParser returns a new FileParser that reads from the open file. The file is not closed.

func (*FileParser) Error

func (p *FileParser) Error() error

Error returns an error, if any, encountered while parsing the slow log.

func (*FileParser) Events

func (p *FileParser) Events() <-chan Event

Events returns the channel to which events from the slow log are sent. The channel is closed when there are no more events. Events are not sent until Start is called.

func (*FileParser) Start

func (p *FileParser) Start(opt Options) error

Start starts the parser. Events are sent to the unbuffered Events channel. Parsing stops on EOF, error, or call to Stop. The Events channel is closed when parsing stops.

func (*FileParser) Stop

func (p *FileParser) Stop()

Stop stops the parser before parsing the next event or while blocked on sending the current event to the event channel.

type Metrics

type Metrics struct {
	TimeMetrics   map[string]*TimeStats   `json:",omitempty"`
	NumberMetrics map[string]*NumberStats `json:",omitempty"`
	BoolMetrics   map[string]*BoolStats   `json:",omitempty"`
}

Metrics encapsulate the metrics of an event like Query_time and Rows_sent.

func NewMetrics

func NewMetrics() Metrics

NewMetrics returns a pointer to an initialized Metrics structure.

func (*Metrics) AddEvent

func (m *Metrics) AddEvent(e Event, outlier bool)

AddEvent saves all the metrics of the event.

func (*Metrics) Finalize

func (m *Metrics) Finalize(rateLimit uint)

Finalize calculates the statistics of the added metrics. Call this function when done adding events.

type NumberStats

type NumberStats struct {
	Sum uint64
	Min uint64 `json:",omitempty"`
	Avg uint64 `json:",omitempty"`
	Med uint64 `json:",omitempty"` // median
	P95 uint64 `json:",omitempty"` // 95th percentile
	Max uint64 `json:",omitempty"`
	// contains filtered or unexported fields
}

NumberStats are integer-based metrics like Rows_sent and Merge_passes.

type Options

type Options struct {
	StartOffset        uint64          // byte offset in file at which to start parsing
	FilterAdminCommand map[string]bool // admin commands to ignore
}

Options encapsulate common options for making a new LogParser.

type Parser

type Parser interface {
	Start(Options) error
	Events() <-chan Event
	Stop()
	Error() error
}

A Parser parses events from a slow log. The canonical Parser is FileParser because the slow log is a file. The caller receives events on the Events channel. This channel is closed when there are no more events. Any error during parsing is returned by Error.

type Result

type Result struct {
	Global    *Class            // all classes
	Class     map[string]*Class // keyed on class ID
	RateLimit uint
	Error     string
}

A Result contains a global class and per-ID classes with finalized metric statistics. The classes are keyed on class ID.

type TimeStats

type TimeStats struct {
	Sum float64
	Min float64 `json:",omitempty"`
	Avg float64 `json:",omitempty"`
	Med float64 `json:",omitempty"` // median
	P95 float64 `json:",omitempty"` // 95th percentile
	Max float64 `json:",omitempty"`
	// contains filtered or unexported fields
}

TimeStats are microsecond-based metrics like Query_time and Lock_time.