README

MySQL Slow Log Parsing and Aggregation

Go Report Card Build Status GoDoc

This package provides functions for working with the MySQL slow log.

Acknowledgement

This code was originally copied from percona/go-mysql @ 2a6037d7d809b18ebd6d735b397f2321879af611. See that project for original contributors and copyright.

This project is a fork to continue development of percona/go-mysql as separate packages. GitHub only allows forking a project once, which is why the code has been copied.

Documentation

Overview

    Package slowlog provides functions and data structures for working with the MySQL slow log.

    Index

    Constants

    View Source
    const (
    	// MAX_EXAMPLE_BYTES defines the maximum Example.Query size.
    	MAX_EXAMPLE_BYTES = 1024 * 10
    )

    Variables

    View Source
    var Debug = false
    View Source
    var (
    	// ErrStarted is returned if Parser.Start is called more than once.
    	ErrStarted = errors.New("parser is started")
    )

    Functions

    This section is empty.

    Types

    type Aggregator

    type Aggregator struct {
    	// contains filtered or unexported fields
    }

      An Aggregator groups events by class ID. When there are no more events, a call to Finalize computes all metric statistics and returns a Result.

      func NewAggregator

      func NewAggregator(samples bool, utcOffset time.Duration, outlierTime float64) *Aggregator

        NewAggregator returns a new Aggregator.

        func (*Aggregator) AddEvent

        func (a *Aggregator) AddEvent(event Event, id, fingerprint string)

          AddEvent adds the event to the aggregator, automatically creating new classes as needed.

          func (*Aggregator) Finalize

          func (a *Aggregator) Finalize() Result

            Finalize calculates all metric statistics and returns a Result. Call this function when done adding events to the aggregator.

            type BoolStats

            type BoolStats struct {
            	Sum uint64 // %true = Sum/Cnt
            	// contains filtered or unexported fields
            }

              BoolStats are boolean-based metrics like QC_Hit and Filesort.

              type Class

              type Class struct {
              	Id            string   // 32-character hex checksum of fingerprint
              	Fingerprint   string   // canonical form of query: values replaced with "?"
              	Metrics       Metrics  // statistics for each metric, e.g. max Query_time
              	TotalQueries  uint64   // total number of queries in class
              	UniqueQueries uint     // unique number of queries in class
              	Example       *Example `json:",omitempty"` // sample query with max Query_time
              
              	MaxQueryCommentMetadata map[string]string
              	// contains filtered or unexported fields
              }

                A Class represents all events with the same fingerprint and class ID. This is only enforced by convention, so be careful not to mix events from different classes.

                func NewAggregateClass

                func NewAggregateClass(id, fingerprint string, members []*Class) *Class

                  NewAggregateClass makes a new Class from the given member classes.

                  func NewClass

                  func NewClass(id, fingerprint string, sample bool) *Class

                    NewClass returns a new Class for the class ID and fingerprint. If sample is true, the query with the greatest Query_time is saved.

                    func (*Class) AddEvent

                    func (c *Class) AddEvent(e Event, outlier bool)

                      AddEvent adds an event to the query class.

                      func (*Class) Finalize

                      func (c *Class) Finalize(rateLimit uint)

                        Finalize calculates all metric statistics. Call this function when done adding events to the class.

                        type Event

                        type Event struct {
                        	Offset          uint64 // byte offset in file at which event starts
                        	Ts              string // raw timestamp of event
                        	Admin           bool   // true if Query is admin command
                        	Query           string // SQL query or admin command
                        	User            string
                        	Host            string
                        	Db              string
                        	TimeMetrics     map[string]float64 // *_time and *_wait metrics
                        	NumberMetrics   map[string]uint64  // most metrics
                        	BoolMetrics     map[string]bool    // yes/no metrics
                        	RateType        string             // Percona Server rate limit type
                        	RateLimit       uint               // Percona Server rate limit value
                        	CommentMetadata map[string]string
                        }

                          An Event is a query like "SELECT col FROM t WHERE id = 1", some metrics like Query_time (slow log) or SUM_TIMER_WAIT (Performance Schema), and other metadata like default database, timestamp, etc. Metrics and metadata are not guaranteed to be defined--and frequently they are not--but at minimum an event is expected to define the query and Query_time metric. Other metrics and metadata vary according to MySQL version, distro, and configuration.

                          func NewEvent

                          func NewEvent() *Event

                            NewEvent returns a new Event with initialized metric maps.

                            type Example

                            type Example struct {
                            	QueryTime float64 // Query_time
                            	Db        string  // Schema: <db> or USE <db>
                            	Query     string  // truncated to MAX_EXAMPLE_BYTES
                            	Ts        string  `json:",omitempty"` // in MySQL time zone
                            }

                              A Example is a real query and its database, timestamp, and Query_time. If the query is larger than MAX_EXAMPLE_BYTES, it is truncated and "..." is appended.

                              type FileParser

                              type FileParser struct {
                              	*sync.Mutex
                              	// contains filtered or unexported fields
                              }

                                FileParser represents a file-based Parser. This is the canonical Parser because the slow log is a file.

                                func NewFileParser

                                func NewFileParser(file *os.File) *FileParser

                                  NewFileParser returns a new FileParser that reads from the open file. The file is not closed.

                                  func (*FileParser) Error

                                  func (p *FileParser) Error() error

                                    Error returns an error, if any, encountered while parsing the slow log.

                                    func (*FileParser) Events

                                    func (p *FileParser) Events() <-chan Event

                                      Events returns the channel to which events from the slow log are sent. The channel is closed when there are no more events. Events are not sent until Start is called.

                                      func (*FileParser) Start

                                      func (p *FileParser) Start(opt Options) error

                                        Start starts the parser. Events are sent to the unbuffered Events channel. Parsing stops on EOF, error, or call to Stop. The Events channel is closed when parsing stops.

                                        func (*FileParser) Stop

                                        func (p *FileParser) Stop()

                                          Stop stops the parser before parsing the next event or while blocked on sending the current event to the event channel.

                                          type Metrics

                                          type Metrics struct {
                                          	TimeMetrics   map[string]*TimeStats   `json:",omitempty"`
                                          	NumberMetrics map[string]*NumberStats `json:",omitempty"`
                                          	BoolMetrics   map[string]*BoolStats   `json:",omitempty"`
                                          }

                                            Metrics encapsulate the metrics of an event like Query_time and Rows_sent.

                                            func NewMetrics

                                            func NewMetrics() Metrics

                                              NewMetrics returns a pointer to an initialized Metrics structure.

                                              func (*Metrics) AddEvent

                                              func (m *Metrics) AddEvent(e Event, outlier bool)

                                                AddEvent saves all the metrics of the event.

                                                func (*Metrics) Finalize

                                                func (m *Metrics) Finalize(rateLimit uint)

                                                  Finalize calculates the statistics of the added metrics. Call this function when done adding events.

                                                  type NumberStats

                                                  type NumberStats struct {
                                                  	Sum uint64
                                                  	Min uint64 `json:",omitempty"`
                                                  	Avg uint64 `json:",omitempty"`
                                                  	Med uint64 `json:",omitempty"` // median
                                                  	P95 uint64 `json:",omitempty"` // 95th percentile
                                                  	Max uint64 `json:",omitempty"`
                                                  	// contains filtered or unexported fields
                                                  }

                                                    NumberStats are integer-based metrics like Rows_sent and Merge_passes.

                                                    type Options

                                                    type Options struct {
                                                    	StartOffset        uint64          // byte offset in file at which to start parsing
                                                    	FilterAdminCommand map[string]bool // admin commands to ignore
                                                    }

                                                      Options encapsulate common options for making a new LogParser.

                                                      type Parser

                                                      type Parser interface {
                                                      	Start(Options) error
                                                      	Events() <-chan Event
                                                      	Stop()
                                                      	Error() error
                                                      }

                                                        A Parser parses events from a slow log. The canonical Parser is FileParser because the slow log is a file. The caller receives events on the Events channel. This channel is closed when there are no more events. Any error during parsing is returned by Error.

                                                        type Result

                                                        type Result struct {
                                                        	Global    *Class            // all classes
                                                        	Class     map[string]*Class // keyed on class ID
                                                        	RateLimit uint
                                                        	Error     string
                                                        }

                                                          A Result contains a global class and per-ID classes with finalized metric statistics. The classes are keyed on class ID.

                                                          type TimeStats

                                                          type TimeStats struct {
                                                          	Sum float64
                                                          	Min float64 `json:",omitempty"`
                                                          	Avg float64 `json:",omitempty"`
                                                          	Med float64 `json:",omitempty"` // median
                                                          	P95 float64 `json:",omitempty"` // 95th percentile
                                                          	Max float64 `json:",omitempty"`
                                                          	// contains filtered or unexported fields
                                                          }

                                                            TimeStats are microsecond-based metrics like Query_time and Lock_time.