bench

package
Version: v0.0.0-...-abed277 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 19, 2019 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Overview

bench contains benchmarks and common utilities useful to benchmarks

In order to write new benchmarks, one must satisfy the Benchmark interface in bench.go. In order to use the benchmark from pi, it needs a new file under tools/cmd which defines a cobra.Cmd - look at an existing benchmark's file to see what needs to be done.

When writing a new benchmark, there are a few things to keep in mind other than just implementing the interface:

1. The benchmark should modify its own configuration in its Init method based on the agentNum it is given. How it modifies is specific to the benchmark, but the idea is that it should make sense to call the benchmark with the same configuration, but multiple different agent numbers, and it should do useful work each time (i.e. not just setting the same bits, or running the same queries).

2. The Init method should do everything that needs to be done to get the benchmark to a runnable state - all code in run should be the stuff that we actually want to time.

3. The Run method does not need to report the total runtime - that is collected by calling code.

Files:

1. client.go contains pilosa client code which is shared by many benchmarks

2. errgroup.go contains the ErrGroup implementation copied from golang.org/x/ so as not to pull in a bunch of useless deps.

3. stats.go contains useful code for gathering stats about a series of timed operations.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Prettify

func Prettify(m map[string]interface{}) map[string]interface{}

Recursively replaces elements of ugly types with their pretty wrappers

Types

type BasicQueryBenchmark

type BasicQueryBenchmark struct {
	Name       string `json:"name"`
	MinRowID   int64  `json:"min-row-id"`
	Iterations int    `json:"iterations"`
	NumArgs    int    `json:"num-args"`
	Query      string `json:"query"`
	Index      string `json:"index"`
	Field      string `json:"field"`

	Logger *log.Logger `json:"-"`
}

BasicQueryBenchmark runs a query multiple times with increasing row ids.

func NewBasicQueryBenchmark

func NewBasicQueryBenchmark() *BasicQueryBenchmark

NewBasicQueryBenchmark returns a new instance of BasicQueryBenchmark.

func (*BasicQueryBenchmark) Run

func (b *BasicQueryBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark.

type Benchmark

type Benchmark interface {
	Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)
}

Benchmark is an interface run benchmark components. Benchmarks should Marshal to valid JSON so that their configuration may be recorded with their results.

type DiagonalSetBitsBenchmark

type DiagonalSetBitsBenchmark struct {
	Name        string `json:"name"`
	MinRowID    int    `json:"min-row-id"`
	MinColumnID int    `json:"min-column-id"`
	Iterations  int    `json:"iterations"`
	Index       string `json:"index"`
	Field       string `json:"field"`

	Logger *log.Logger `json:"-"`
}

DiagonalSetBitsBenchmark sets bits with increasing column id and row id.

func NewDiagonalSetBitsBenchmark

func NewDiagonalSetBitsBenchmark() *DiagonalSetBitsBenchmark

NewDiagonalSetBitsBenchmark returns a new instance of DiagonalSetBitsBenchmark.

func (*DiagonalSetBitsBenchmark) Run

func (b *DiagonalSetBitsBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark.

type HostSetup

type HostSetup struct {
	Hosts         []string
	ClientOptions []pilosa.ClientOption
}

type ImportBenchmark

type ImportBenchmark struct {
	Name         string `json:"name"`
	MinRowID     int64  `json:"min-row-id"`
	MinColumnID  int64  `json:"min-column-id"`
	MaxRowID     int64  `json:"max-row-id"`
	MaxColumnID  int64  `json:"max-column-id"`
	Index        string `json:"index"`
	Field        string `json:"field"`
	Iterations   int64  `json:"iterations"`
	Seed         int64  `json:"seed"`
	Distribution string `json:"distribution"`
	BufferSize   int    `json:"-"`

	Logger *log.Logger `json:"-"`
}

func NewImportBenchmark

func NewImportBenchmark() *ImportBenchmark

NewImportBenchmark returns a new instance of ImportBenchmark.

func (*ImportBenchmark) RecordIterator

func (b *ImportBenchmark) RecordIterator(seed int64) *RecordIterator

func (*ImportBenchmark) Run

func (b *ImportBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the Import benchmark

type ImportRangeBenchmark

type ImportRangeBenchmark struct {
	Name         string `json:"name"`
	MinValue     int64  `json:"min-value"`
	MinColumnID  int64  `json:"min-column-id"`
	MaxValue     int64  `json:"max-value"`
	MaxColumnID  int64  `json:"max-column-id"`
	Index        string `json:"index"`
	Field        string `json:"field"`
	Row          string `json:"row"`
	Iterations   int64  `json:"iterations"`
	Seed         int64  `json:"seed"`
	Distribution string `json:"distribution"`
	BufferSize   int    `json:"-"`

	Logger *log.Logger `json:"-"`
}

func NewImportRangeBenchmark

func NewImportRangeBenchmark() *ImportRangeBenchmark

NewImportRangeBenchmark returns a new instance of ImportRangeBenchmark.

func (*ImportRangeBenchmark) Run

func (b *ImportRangeBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark.

func (*ImportRangeBenchmark) ValueIterator

func (b *ImportRangeBenchmark) ValueIterator(seed int64) *ValueIterator

type NumStats

type NumStats struct {

	// NumZero counts the number of values that have been added which
	// are zero. This is a cheap, simple, replacement for more
	// sophisticated tracking of the distribution of the data that
	// let's us know if (e.g.) we have a bunch of queries doing
	// nothing because we're querying empty rows or something.
	NumZero int64   `json:"num-zero"`
	Min     int64   `json:"min"`
	Max     int64   `json:"max"`
	Mean    int64   `json:"mean"`
	Total   int64   `json:"total"`
	Num     int64   `json:"num"`
	All     []int64 `json:"all"`
	SaveAll bool    `json:"-"`
	// contains filtered or unexported fields
}

NumStats object helps track stats. This and Stats (which was originally made specifically for time) should probably be unified.

func NewNumStats

func NewNumStats() *NumStats

NewNumStats gets a NumStats object.

func (*NumStats) Add

func (s *NumStats) Add(td int64)

Add adds a new value to the stats object.

func (*NumStats) Combine

func (s *NumStats) Combine(other *NumStats)

type PrettyDuration

type PrettyDuration time.Duration

wrapper type to force human-readable JSON output

func (PrettyDuration) MarshalJSON

func (d PrettyDuration) MarshalJSON() ([]byte, error)

MarshalJSON returns a nicely formatted duration, instead of it just being treated like an int.

type QueryBenchmark

type QueryBenchmark struct {
	Name       string `json:"name"`
	Query      string `json:"query"`
	Index      string `json:"index"`
	Iterations int    `json:"iterations"`

	Logger *log.Logger `json:"-"`
}

func NewQueryBenchmark

func NewQueryBenchmark() *QueryBenchmark

func (*QueryBenchmark) Run

func (b *QueryBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark.

type QueryGenerator

type QueryGenerator struct {
	// contains filtered or unexported fields
}

QueryGenerator holds the configuration and state for randomly generating queries.

func NewQueryGenerator

func NewQueryGenerator(index *pilosa.Index, field *pilosa.Field, seed int64) *QueryGenerator

NewQueryGenerator returns a new QueryGenerator.

func (*QueryGenerator) Random

func (g *QueryGenerator) Random(maxN, depth, maxargs int, idmin, idmax uint64) pilosa.PQLQuery

Random returns a randomly generated query.

func (*QueryGenerator) RandomBitmapCall

func (g *QueryGenerator) RandomBitmapCall(depth, maxargs int, idmin, idmax uint64) *pilosa.PQLRowQuery

RandomBitmapCall returns a randomly generate query which returns a bitmap.

func (*QueryGenerator) RandomRange

func (g *QueryGenerator) RandomRange(numArg int, idmin, idmax uint64) *pilosa.PQLRowQuery

func (*QueryGenerator) RandomRangeQuery

func (g *QueryGenerator) RandomRangeQuery(depth, maxargs int, idmin, idmax uint64) pilosa.PQLQuery

RandomRangeQuery returns a randomly generated sum or range query.

func (*QueryGenerator) RandomSum

func (g *QueryGenerator) RandomSum(depth, maxargs int, idmin, idmax uint64) pilosa.PQLQuery

RandomSum returns a randomly generated sum query.

func (*QueryGenerator) RandomTopN

func (g *QueryGenerator) RandomTopN(maxN, depth, maxargs int, idmin, idmax uint64) *pilosa.PQLRowQuery

RandomTopN returns a randomly generated TopN query.

func (*QueryGenerator) RangeCall

func (g *QueryGenerator) RangeCall(idmin, idmax uint64) *pilosa.PQLRowQuery

type RandomQueryBenchmark

type RandomQueryBenchmark struct {
	Name       string `json:"name"`
	MaxDepth   int    `json:"max-depth"`
	MaxArgs    int    `json:"max-args"`
	MaxN       int    `json:"max-n"`
	MinRowID   int64  `json:"min-row-id"`
	MaxRowID   int64  `json:"max-row-id"`
	Iterations int    `json:"iterations"`
	Seed       int64  `json:"seed"`
	Index      string `json:"index"`
	Field      string `json:"field"`

	Logger *log.Logger `json:"-"`
}

RandomQueryBenchmark queries randomly and deterministically based on a seed.

func NewRandomQueryBenchmark

func NewRandomQueryBenchmark() *RandomQueryBenchmark

NewRandomQueryBenchmark returns a new instance of RandomQueryBenchmark.

func (*RandomQueryBenchmark) Run

func (b *RandomQueryBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the RandomQuery benchmark

type RandomSetBenchmark

type RandomSetBenchmark struct {
	Name          string `json:"name"`
	MinRowID      int64  `json:"min-row-id"`
	MaxRowID      int64  `json:"max-row-id"`
	MinColumnID   int64  `json:"min-column-id"`
	MaxColumnID   int64  `json:"max-column-id"`
	Iterations    int    `json:"iterations"`
	BatchSize     int    `json:"batch-size"`
	Seed          int64  `json:"seed"`
	NumAttrs      int    `json:"num-attrs"`
	NumAttrValues int    `json:"num-attr-values"`
	Index         string `json:"index"`
	Field         string `json:"field"`

	Logger *log.Logger `json:"-"`
}

RandomSetBenchmark sets bits randomly and deterministically based on a seed.

func NewRandomSetBenchmark

func NewRandomSetBenchmark() *RandomSetBenchmark

NewRandomSetBenchmark returns a new instance of RandomSetBenchmark.

func (*RandomSetBenchmark) Run

func (b *RandomSetBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark.

type RangeQueryBenchmark

type RangeQueryBenchmark struct {
	Name       string `json:"name"`
	MaxDepth   int    `json:"max-depth"`
	MaxArgs    int    `json:"max-args"`
	MaxN       int    `json:"max-n"`
	MinRange   int64  `json:"min-range"`
	MaxRange   int64  `json:"max-range"`
	Iterations int    `json:"iterations"`
	Seed       int64  `json:"seed"`
	Frame      string `json:"frame"`
	Index      string `json:"index"`
	Field      string `json:"field"`
	QueryType  string `json:"type"`

	Logger *log.Logger `json:"-"`
}

RangeQueryBenchmark runs Range query randomly.

func NewRangeQueryBenchmark

func NewRangeQueryBenchmark() *RangeQueryBenchmark

NewRangeQueryBenchmark returns a new instance of RangeQueryBenchmark.

func (*RangeQueryBenchmark) Run

func (b *RangeQueryBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark.

type RecordIterator

type RecordIterator struct {
	// contains filtered or unexported fields
}

func NewRecordIterator

func NewRecordIterator() *RecordIterator

func (*RecordIterator) NextRecord

func (itr *RecordIterator) NextRecord() (pilosa.Record, error)

type Result

type Result struct {
	Stats         *Stats                  `json:"stats"`
	Responses     []*pilosa.QueryResponse `json:"responses"`
	Extra         map[string]interface{}  `json:"extra"`
	AgentNum      int                     `json:"agentnum"`
	PilosaVersion string                  `json:"pilosa-version"`
	Configuration interface{}             `json:"configuration"`

	// Error exists so that errors can be correctly marshalled to JSON. It is set using Result.err.Error()
	Error string `json:"error,omitempty"`
}

Results holds the output from the run of a benchmark - the Benchmark's Run() method may set Stats, Responses, and Extra, and the RunBenchmark helper function will set the Duration, AgentNum, PilosaVersion, and Configuration. Either may set Error if there is an error. The structure of Result assumes that most benchmarks will run multiple queries and track statistics about how long each one takes. The Extra field is for benchmarks which either do not fit this model, or want to return additional information not covered by Stats and Responses.

func NewResult

func NewResult() *Result

NewResult intializes and returns a Result.

func (*Result) Add

func (r *Result) Add(d time.Duration, resp *pilosa.QueryResponse)

Add adds the duration to the Result's Stats object. If resp is non-nil, it also adds it to the slice of responses.

type SliceWidthBenchmark

type SliceWidthBenchmark struct {
	Name       string  `json:"name"`
	Index      string  `json:"index"`
	Field      string  `json:"field"`
	BitDensity float64 `json:"bit-density"`
	SliceWidth int64   `json:"slice-width"`
	SliceCount int64   `json:"slice-count"`

	Logger *log.Logger `json:"-"`
}

SliceWidthBenchmark helps importing data based on slice-width and data density. a single slice on query time.

func NewSliceWidthBenchmark

func NewSliceWidthBenchmark() *SliceWidthBenchmark

NewSliceWidthBenchmark creates slice width benchmark.

func (*SliceWidthBenchmark) Run

func (b *SliceWidthBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark to import data.

type Stats

type Stats struct {
	Min     time.Duration   `json:"min"`
	Max     time.Duration   `json:"max"`
	Mean    time.Duration   `json:"mean"`
	Total   time.Duration   `json:"total-time"`
	Num     int64           `json:"num"`
	All     []time.Duration `json:"all"`
	SaveAll bool            `json:"-"`
	// contains filtered or unexported fields
}

Stats object helps track timing stats.

func NewStats

func NewStats() *Stats

NewStats gets a Stats object.

func (*Stats) Add

func (s *Stats) Add(td time.Duration)

Add adds a new time to the stats object.

func (*Stats) Combine

func (s *Stats) Combine(other *Stats)

type TPSBenchmark

type TPSBenchmark struct {
	Name        string   `json:"name"`
	Intersect   bool     `json:"intersect" help:"If true, include Intersect queries in benchmark."`
	Union       bool     `json:"union" help:"If true, include Union queries in benchmark."`
	Difference  bool     `json:"difference" help:"If true, include Difference queries in benchmark."`
	Xor         bool     `json:"xor" help:"If true, include XOR queries in benchmark."`
	Fields      []string `json:"fields" help:"Comma separated list of fields. If blank, use all fields in index schema."`
	MinRowID    int64    `json:"min-row-id" help:"Minimum row ID to use in queries."`
	MaxRowID    int64    `json:"max-row-id" help:"Max row ID to use in queries. If 0, determine max available."`
	Index       string   `json:"index" help:"Index to use. If blank, one is chosen randomly from the schema."`
	Concurrency int      `json:"concurrency" help:"Run this many goroutines concurrently." short:"y"`
	Iterations  int      `json:"iterations" help:"Each goroutine will perform this many queries."`

	Logger *log.Logger `json:"-"`
}

func NewTPSBenchmark

func NewTPSBenchmark() *TPSBenchmark

func (*TPSBenchmark) Run

func (b *TPSBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the benchmark.

type ValueIterator

type ValueIterator struct {
	// contains filtered or unexported fields
}

func NewValueIterator

func NewValueIterator() *ValueIterator

func (*ValueIterator) NextRecord

func (itr *ValueIterator) NextRecord() (pilosa.Record, error)

type ZipfBenchmark

type ZipfBenchmark struct {
	Name           string  `json:"name"`
	MinRowID       int64   `json:"min-row-id"`
	MinColumnID    int64   `json:"min-column-id"`
	MaxRowID       int64   `json:"max-row-id"`
	MaxColumnID    int64   `json:"max-column-id"`
	Iterations     int     `json:"iterations"`
	Seed           int64   `json:"seed"`
	Index          string  `json:"index"`
	Field          string  `json:"field"`
	RowExponent    float64 `json:"row-exponent"`
	RowRatio       float64 `json:"row-ratio"`
	ColumnExponent float64 `json:"column-exponent"`
	ColumnRatio    float64 `json:"column-ratio"`
	Operation      string  `json:"operation"`

	Logger *log.Logger `json:"-"`
}

ZipfBenchmark sets random bits according to the Zipf-Mandelbrot distribution. This distribution accepts two parameters, Exponent and Ratio, for both rows and columns. It also uses PermutationGenerator to permute IDs randomly.

func NewZipfBenchmark

func NewZipfBenchmark() *ZipfBenchmark

NewZipfBenchmark returns a new instance of ZipfBenchmark.

func (*ZipfBenchmark) Run

func (b *ZipfBenchmark) Run(ctx context.Context, client *pilosa.Client, agentNum int) (*Result, error)

Run runs the Zipf benchmark

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL