optimize

Documentation

Overview

Package optimize provides a set of services for optimizing algo parameters. Parameter optimization is a process of systematically searching for the optimal set of parameters given a target objective. Multiple methods are available, each implementing the Optimizer interface.

Example

// Verbose error handling omitted for brevity

// Identify the bot (algo) to optimize by supplying a factory function
// Here we're using the classic moving average (MA) cross variant of trend bot
bot := trend.MakeCrossBotFromConfig

// Define the parameter space to optimize
// Param names must match those expected by the MakeBot function passed to optimizer
// Here we're optimizing the lookback period of a fast and slow MA
// and the Market Meanness Index (MMI) filter
paramSpace := ParamMap{
	"mafastlength": []any{30, 90, 180},
	"maslowlength": []any{90, 180, 360},
	"mmilength":    []any{200, 300},
}

// Read price samples to use for optimization
btc, err := market.ReadKlinesFromCSV("./testdata/BTCUSDT-1H/")
if err != nil {
	log.Fatal(err)
}
eth, err := market.ReadKlinesFromCSV("./testdata/ETHUSDT-1H/")
if err != nil {
	log.Fatal(err)
}
priceSamples := map[AssetID][]market.Kline{"btc": btc, "eth": eth}

// Create a new brute style optimizer with a default simulated dealer (no broker costs)
// The default optimization objective is the param set with the highest sharpe ratio
optimizer := NewBruteOptimizer()
optimizer.SampleSplitPct = 0   // Do not split samples due to small sample size
optimizer.WarmupBarCount = 360 // Set as maximum lookback of your param space
optimizer.MakeBot = bot        // Tell the optimizer which bot to use

// Prepare the optimizer and get an estimate on the number of trials (backtests) required
trialCount, _ := optimizer.Prepare(paramSpace, priceSamples)
fmt.Printf("%d backtest trials to run during optimization\n", trialCount)

// Start the optimization process and monitor with a receive-only channel
// Trials will execute concurrently with a default worker pool matching num of CPUs
trials, _ := optimizer.Start(context.Background())
for range trials {
}

// Inspect the study results following optimization
study := optimizer.Study()
if len(study.ValidationResults) == 0 {
	fmt.Println("Optima not found because highest ranked param set made no trades during optimization trials.")
	return
}

// Read out the optimal param set and results
optimaPSet := study.Validation[0]
fmt.Printf("Optima params: fast: %d slow: %d MMI: %d\n",
	optimaPSet.Params["mafastlength"], optimaPSet.Params["maslowlength"], optimaPSet.Params["mmilength"])
optimaResult := study.ValidationResults[optimaPSet.ID]
fmt.Printf("Optima sharpe ratio is %.1f", optimaResult.Sharpe)

Output:

38 backtest trials to run during optimization
Optima params: fast: 30 slow: 90 MMI: 200
Optima sharpe ratio is 2.5

Index

• func PRRRanker(a, b PhaseReport) bool
• func SharpeRanker(a, b PhaseReport) bool
• func WriteStudyResultToCSV(path string, study *Study) error
• type AssetID
• type BruteOptimizer
  • func NewBruteOptimizer() BruteOptimizer
  • func (o *BruteOptimizer) Prepare(in ParamMap, samples map[AssetID][]market.Kline) (int, error)
  • func (o *BruteOptimizer) Start(ctx context.Context) (<-chan OptimizerTrial, error)
  • func (o *BruteOptimizer) Study() *Study
• type CartesianProduct
  • func CartesianBuilder(in map[string]any) []CartesianProduct
• type ObjectiveRanker
• type Optimizer
• type OptimizerTrial
• type ParamMap
• type ParamSet
  • func NewParamSet() ParamSet
• type ParamSetID
• type Phase
  • func (p Phase) MarshalText() ([]byte, error)
  • func (p Phase) String() string
• type PhaseReport
  • func NewReport() PhaseReport
  • func Summarize(report PhaseReport) PhaseReport
• type Study
  • func NewStudy() *Study

Examples

• Package

Functions

func PRRRanker

func PRRRanker(a, b PhaseReport) bool

PRRRanker ranks by Pessimistic Return Ratio.

func SharpeRanker

func SharpeRanker(a, b PhaseReport) bool

SharpeRanker ranks by Sharpe ratio.

func WriteStudyResultToCSV

func WriteStudyResultToCSV(path string, study *Study) error

WriteStudyResultToCSV writes the results of a study to CSV. Study results are flattened to separate files with foreign keys added to link rows. Study ID is prefixed to the result filenames.

- phasereport.csv: summary performance of each paramset for each phase

- trialreport.csv: detailed performance of each trial (backtest) in a phase

- roundturns.csv: turns completed for each trial

- curve.csv: equity curve for each trial

Types

type AssetID

type AssetID string

AssetID is a string identifer for the asset associated with a sample. Typically the symbol of the asset, e.g. btcusdt.

type BruteOptimizer

type BruteOptimizer struct {
	SampleSplitPct float64
	WarmupBarCount int
	MakeBot        trader.MakeFromConfig
	MakeDealer     broker.MakeSimulatedDealer
	Ranker         ObjectiveRanker

	MaxWorkers int
	// contains filtered or unexported fields
}

BruteOptimizer implements a 'brute-force peak objective' optimization approach which tests all given parameter combinations and selects the highest ranked (peak) param set. Optima is selected by the given ObjectiveRanker func. Optimization trials are executed concurrently using a worker pool. Optimization method in 3 stages:

Prepare:

- Accept 1 or more price data samples

- Split sample price data into in-sample (training) and out-of-sample (validation) datasets

- Generate 1 or more param sets using the cartesian product of given ranges that define the param space

Train:

- Execute each algo param set over the in-sample price data

- Average the performance for each param set over the in-sample data

- Rank the param sets based on the performance objective (Profit Factor, Sharpe etc)

Validate:

- Execute the highest ranked ("trained") algo param set over the out-of-sample price data

- Accept or reject the hypothesis based on statistical significance of the study report

func NewBruteOptimizer

func NewBruteOptimizer() BruteOptimizer

NewBruteOptimizer creates a new BruteOptimizer instance with sensible defaults. Call Prepare before Start to set up the study.

func (*BruteOptimizer) Prepare

func (o *BruteOptimizer) Prepare(in ParamMap, samples map[AssetID][]market.Kline) (int, error)

Prepare prepares a study based on the given param ranges and price data samples. Returned is the estimated number of trials to be performed.

func (*BruteOptimizer) Start

func (o *BruteOptimizer) Start(ctx context.Context) (<-chan OptimizerTrial, error)

Start starts the prepared optimization process and returns with a channel to monitor the progress.

func (*BruteOptimizer) Study

func (o *BruteOptimizer) Study() *Study

Study returns the current study. Call after the optimizer has finished to read the results.

type CartesianProduct

type CartesianProduct map[string]any

CartesianProduct is a map of named params returned by CartesianBuilder.

func CartesianBuilder

func CartesianBuilder(in map[string]any) []CartesianProduct

CartesianBuilder is a utility to build a cartesian product of named params. Used by the BruteOptimizer to find all param combinations.

type ObjectiveRanker

type ObjectiveRanker func(a, b PhaseReport) bool

ObjectiveRanker is used by an Optimizer to sort the results of backtest trials and select the best performing ParamSet. ObjectiveRanker is equivalent to a 'less' comparison function.

type Optimizer

type Optimizer interface {
	Prepare(ParamMap, map[AssetID][]market.Kline) (int, error)
	Start(context.Context) (<-chan OptimizerTrial, error)
	Study() *Study
}

Optimizer is the interface for an optimization method.

type OptimizerTrial

type OptimizerTrial struct {
	Phase  Phase
	PSet   ParamSet
	Result perf.PerformanceReport
	Err    error
}

OptimizerTrial is a discrete trial conducted by an Optimizer on a single ParamSet.

type ParamMap

type ParamMap map[string]any

ParamMap is a map of algo parameters.

type ParamSet

type ParamSet struct {
	ID     ParamSetID `csv:"id"`
	Params ParamMap   `csv:"params"`
}

ParamSet is a set of algo parameters to trial.

func NewParamSet

func NewParamSet() ParamSet

NewParamSet returns a new param set with initialized ID

type ParamSetID

type ParamSetID string

ParamSetID is a unique identifier for a ParamSet.

type Phase

type Phase int

Phase is the phase of the optimization method.

const (
	// Training is the in-sample phase used to select the optimal parameters.
	Training Phase = iota + 1

	// Validation is the out-of-sample phase used to evaluate the performance of the optimal parameters.
	Validation
)

func (Phase) MarshalText

func (p Phase) MarshalText() ([]byte, error)

MarshalText is used to output the phase as a string for CSV rendering.

func (Phase) String

func (p Phase) String() string

String returns the string representation of the phase.

type PhaseReport

type PhaseReport struct {
	ID      string   `csv:"id"`
	Phase   Phase    `csv:"phase"`
	Subject ParamSet `csv:"paramset_,inline"`

	PRR        float64 `csv:"prr"`
	MDD        float64 `csv:"mdd"`
	CAGR       float64 `csv:"cagr"`
	HistVolAnn float64 `csv:"vol"`
	Sharpe     float64 `csv:"sharpe"`
	Calmar     float64 `csv:"calmar"`
	WinPct     float64 `csv:"win_pct"`

	Kelly    float64 `csv:"kelly"`
	OptimalF float64 `csv:"optimalf"`

	SampleCount    int `csv:"sample_count"`
	RoundTurnCount int `csv:"roundturn_count"`

	Trials []perf.PerformanceReport `csv:"-"`
}

PhaseReport is the aggregated performance of a ParamSet across one or more price samples (trials) The summary method is owned by the Optimizer implementation, but will typically be the mean (avg) of the individual trials.

func NewReport

func NewReport() PhaseReport

NewReport returns a new empty report with an initialized ID.

func Summarize

func Summarize(report PhaseReport) PhaseReport

Summarize inspects the individual backtests in the report and updates the summary fields. Summary takes the average of the metric across all the trials in a phase.

type Study

type Study struct {
	ID string

	Training        []ParamSet
	TrainingSamples map[AssetID][]market.Kline
	TrainingResults map[ParamSetID]PhaseReport

	Validation        []ParamSet
	ValidationSamples map[AssetID][]market.Kline
	ValidationResults map[ParamSetID]PhaseReport
}

Study is an optimization experiment, prepared and executed by an Optimizer. First, a training (in-sample) phase is conducted, followed by a validation (out-of-sample) phase. The validation phase reports the out-of-sample (OOS) performance of the optimum param set.

The experiment can be summarised as:

- Hypothesis: the optimized algo params will generate positive market returns in live trading.

- Null hypothesis: algo has zero positive expectancy of returns in the tested param space.

- Independent variable (aka predictor / feature): algo parameter space defined by []ParamSet.

- Dependent variable: algo backtest performance (Sharpe, CAGR et al) measured by Report.

- Control variables: price data samples, backtest simulator settings etc.

- Method: as defined by the Optimizer implementation (e.g. brute force, genetic et al) and its ObjectiveRanker func.

func NewStudy

func NewStudy() *Study

NewStudy returns a new study. Use an Optimizer implementation to prepare and execute the study.