performance

README

Babelfish performance testing

Build

go generate ./...
cd cmd/bblfsh-performance
go build

Currently supports only 2 commands

parse-and-store

./bblfsh-performance parse-and-store --help
parse file(s) with golang benchmark output and store it into a given storage

Usage:
  bblfsh-performance parse-and-store [--language=<language>] [--commit=<commit-id>] [--storage=<storage>] <file ...> [flags]

Aliases:
  parse-and-store, pas, parse-and-dump

Examples:
WARNING! To access storage corresponding environment variables should be set.
Full examples of usage scripts are following:

# for prometheus pushgateway
export PROM_ADDRESS="localhost:9091"
export PROM_JOB=pushgateway
bblfsh-performance parse-and-store --language=go --commit=3d9682b --storage="prom" /var/log/bench0 /var/log/bench1

# for influx db
export INFLUX_ADDRESS="http://localhost:8086"
export INFLUX_USERNAME=""
export INFLUX_PASSWORD=""
export INFLUX_DB=mydb
export INFLUX_MEASUREMENT=benchmark
bblfsh-performance parse-and-store --language=go --commit=3d9682b --storage="influxdb" /var/log/bench0 /var/log/bench1

Flags:
  -c, --commit string     commit id that's being tested and will be used as a tag in performance report
  -h, --help              help for parse-and-store
  -l, --language string   name of the language to be tested
  -s, --storage string    storage kind to store the results(prom, influxdb, file) (default "prom")

Command usage

Either locally or in CI:

1. pull driver repo
2. perform benchmarks over the fixtures
3. save output test benchmark output to the file
4. run bblfsh-performance parse-and-store and pass the filepath(s) as an argument

end-2-end

./bblfsh-performance end-to-end --help
run bblfshd container and perform benchmark tests, store results into a given storage

Usage:
  bblfsh-performance end-to-end [--language=<language>] [--commit=<commit-id>] [--extension=<files-extension>] [--docker-tag=<docker-tag>] [--storage=<storage>] <directory ...> [flags]

Aliases:
  end-to-end, e2e

Examples:
To use external bblfshd set BBLFSHD_LOCAL=${bblfshd_address}

WARNING! To access storage corresponding environment variables should be set.
Full examples of usage scripts are following:

# for prometheus pushgateway
export PROM_ADDRESS="localhost:9091"
export PROM_JOB=pushgateway
./bblfsh-performance end-to-end --language=go --commit=3d9682b --extension=".go" --storage="prom" /var/testdata/benchmarks

# for influx db
export INFLUX_ADDRESS="http://localhost:8086"
export INFLUX_USERNAME=""
export INFLUX_PASSWORD=""
export INFLUX_DB=mydb
export INFLUX_MEASUREMENT=benchmark
bblfsh-performance end-to-end --language=go --commit=3d9682b --filter-prefix="bench_" --extension=".go" --storage="influxdb" /var/testdata/benchmarks

Flags:
  -c, --commit string          commit id that's being tested and will be used as a tag in performance report
      --custom-driver          if this flag is set to true CLI pulls corresponding language driver repo's commit, builds docker image and installs it onto the bblfsh container
  -t, --docker-tag string      bblfshd docker image tag to be tested (default "latest-drivers")
  -e, --extension string       file extension to be filtered
      --filter-prefix string   file prefix to be filtered (default "bench_")
  -h, --help                   help for end-to-end
  -l, --language string        name of the language to be tested
  -s, --storage string         storage kind to store the results(prom, influxdb, file) (default "prom")

Documentation

Index

• Constants
• func ExecCmd(command string) error
• func GetFiles(pref, ext string, dirs ...string) ([]string, error)
• func SplitStringMap(m map[string]string) ([]string, []string)
• func WrapErr(err error, kinds ...*errors.Kind) error
• type Benchmark
  • func BenchmarkResultToBenchmark(name string, b *testing.BenchmarkResult) Benchmark
  • func NewBenchmark(pb *parse.Benchmark) Benchmark
• type RunE
  • func RunESilenced(f RunE) RunE

Constants

const (
	// BblfshdLevel is a metrics tag that represents benchmarks being run over bblfshd container
	BblfshdLevel = "bblfshd"
	// DriverLevel is a metrics tag that represents benchmarks being run over language driver container
	DriverLevel = "driver"
	// TransformsLevel is a metrics tag that represents benchmarks being run over transformations layer
	TransformsLevel = "transforms"
)

Functions

func ExecCmd

func ExecCmd(command string) error

ExecCmd executes the specified Bash script. If execution fails, the error contains the combined output from stdout and stderr of the script. Do not use this for scripts that produce a large volume of output.

func GetFiles

func GetFiles(pref, ext string, dirs ...string) ([]string, error)

GetFiles is a simple "get files by pattern" function Purpose: filter required fixtures

func SplitStringMap

func SplitStringMap(m map[string]string) ([]string, []string)

SplitStringMap splits map[string]string to arrays of keys and arrays of values

func WrapErr

func WrapErr(err error, kinds ...*errors.Kind) error

WrapErr wraps given error with a given amount of error kinds. Works in inside-to-outside direction

Types

type Benchmark

type Benchmark struct {
	Benchmark parse.Benchmark
}

Benchmark is a wrapper around parse.Benchmark and serves for formatting and arranging data before storing

func BenchmarkResultToBenchmark

func BenchmarkResultToBenchmark(name string, b *testing.BenchmarkResult) Benchmark

BenchmarkResultToBenchmark converts b *testing.BenchmarkResult *parse.Benchmark for further storing

func NewBenchmark

func NewBenchmark(pb *parse.Benchmark) Benchmark

NewBenchmark is a constructor for Benchmark

type RunE

type RunE func(cmd *cobra.Command, args []string) error

RunE is a type that represents a standard Run function for cobra commands

func RunESilenced

func RunESilenced(f RunE) RunE

TODO(lwsanty): https://github.com/spf13/cobra/issues/340 RunESilenced is a wrapper over standard cobra's RunE function Purpose: hide the command usage output in the case of internal error inside the command