gosteps

README

GoSteps - v1

GoSteps is a go library that helps in running functions as steps and reminds you to step out and get active (kidding!).

The idea behind gosteps is to define set of functions as chain-of-steps and execute them in a sequential fashion.

[!NOTE] go-steps v1 is a breaking change and older v0 models won't work with the new version. To continue to use the v0, you can either use the v0 sub-package of the library or the v0.3.0 tag. For usage documentation of v0, refer to the v0 README.

Usage

Functions that needs execution are defined as a Step, and each step is a part of a collection of multiple Steps called Branch, that runs sequentially. Each step can have multiple Branches as possible next steps, and the branch to be executed is determined based on a ResolverFuntion of that step.

Installation

go get github.com/TanmoySG/go-steps

Types

Different types defined in GoSteps are required to create a step chain.

Step

The Step type contains the requirments to execute a step function and is a part of a Branch.

type Step struct {
  Name            StepName               `json:"name"`
  Function        StepFn                 `json:"-"`
  StepOpts        StepOpts               `json:"stepConfig"`
  Branches        *Branches              `json:"branches"`
  StepArgs        map[string]interface{} `json:"stepArgs"`
}

// example step
step := gosteps.Step{
  Name: "add",
  Function: Add,
  StepArgs: map[string]interface{}{"n1": 5},
  StepOpts: gosteps.StepOpts{},
  Branches: &gosteps.Branches{},
}

Field	Description
Name	Name of step
Function	The function to execute
StepOpts	Options/Configurations of the step
Branches	Branches are a sequentially executable collection of steps.
StepArgs	Any additional arguments/variables needed to be passed to the step for execution.

StepOpts

The StepOpts type contains the configurations/options for the step execution.

type StepOpts struct {
  ErrorsToRetry        []error         `json:"errorsToRetry"`
  ErrorPatternsToRetry []regexp.Regexp `json:"errorPatternsToRetry"`
  RetryAllErrors       bool            `json:"retryAllErrors"`
  MaxRunAttempts       int             `json:"maxAttempts"`
  RetrySleep           time.Duration   `json:"retrySleep"`
}

Field	Description
ErrorsToRetry	a set of errors for which a step should be retried.
ErrorPatternsToRetry	a set of StepErrorPatterns for which a step should be retried, if error matches pattern
RetryAllErrors	A boolean type flag which specifies if a step needs to retry for any error, irrespective of those in ErrorsToRetry
MaxRunAttempts	Max attempts are the number of times the step is ran/executed (first run + retries). If not set, it'll run once.
RetrySleep	Sleep duration (type time.Duration) between each re-attempts

Function

Defines a step function of type StepFn.

// StepFn Type
type StepFn func(ctx GoStepsCtx) StepResult

// example function
func Add(ctx GoStepsCtx) StepResult {
  // do something
  return gosteps.MarkStateComplete()
}

Branch

The Branch type is a collection of steps that are to be executed sequentially.

// Steps type - array of Step
type Steps []Step

type Branch struct {
  BranchName BranchName `json:"branchName"`
  Steps      Steps      `json:"steps"`
}

// example branch
branch := gosteps.Branch{
  BranchName: "divide",
  Steps: gosteps.Steps{
   {
    Name: "step.divide",
    Function: func(c gosteps.GoStepsCtx) gosteps.StepResult {
     // do something
     return gosteps.MarkStateComplete()
    },
   },
   { /* other steps*/ },
  },
}

Field	Description
BranchName	Name of the branch
Steps	Steps are a collection of steps that are to be executed sequentially. It is a array of Step ([]Step)

Branches

The Branches type contains an array of branch that are exececuted conditionally, and resolver function that determines the branch to be executed.

type Branches struct {
  Branches []Branch   `json:"branches"`
  Resolver ResolverFn `json:"-"`
}

// example branches
branches := &gosteps.Branches{
  Resolver: ResolverFn,
  Branches: []gosteps.Branch{
   {
    BranchName: "branch1",
    Steps: gosteps.Steps{
     {
      Name:     "branch1.step1",
      Function: StepFn1,
     },
    },
   },
   {
    BranchName: "branch2",
    Steps: gosteps.Steps{
     {
      Name:     "branch2.step1",
      Function: StepFn2,
     },
    },
   },
  },
}

Field	Description
Branches	Branches are a collection of branches that are to be executed conditionally. It is an array of Branch
Resolver	Resolver function to determine the branch to be executed.

ResolverFn

The ResolverFn type is a function that determines the branch to be executed.

// ResolverFn Type
type ResolverFn func(ctx GoStepsCtx) BranchName

// example resolver function
func nextStepResolver(ctx GoStepsCtx) BranchName {
  // do something
  return "branchName"
}

To see all the types available in go-steps see types.go

GoStep Context

GoStepsCtx is a custom context type, instance of which is passed between steps to store and share data, progress, state, etc between steps and branches.

type GoStepsCtx struct {
  Data          GoStepsCtxData            `json:"data"`
  StepsProgress map[StepName]StepProgress `json:"stepsProgress"`
}

// creating a new context
ctx := gosteps.NewGoStepsContext()

Field	Description
Data	Data is a map of key-value pairs that can be used to store and share data between steps and branches
StepsProgress	StepsProgress is a map of StepName and StepProgress that stores the progress of each step

GoStepCtx functions: SetData, GetData and WithData

To set data in the context, use the SetData method.

ctx.SetData("key", "value")

To get data from the context, use the GetData method.

value := ctx.GetData("key")

To set multiple key-value pair in the context, use the WithData method that expects K-V data to be of type map[string]interface{}.

ctx.WithData(map[string]interface{}{
 "key1": "value1",
 "key2": "value2",
})

These functions can be used within Step Functions and Resolver Functions to store and retrieve data and use them in the execution.

Defining an Executable Step Chain

To define a step chain, create a Branch with the list of Steps to be run sequentially. Use the gosteps.Steps type or []gosteps.Step type to define the steps and pass it to the NewGoStepsRunner method, which returns the executable step chain instance.

root := gosteps.NewGoStepsRunner(steps)

To execute the step chain use the Execute method, passing the context.

ctx := gosteps.NewGoStepsContext()
root.Execute(ctx)

You can initialize GoStepsCtx with data using the WithData method, too.

ctx := gosteps.NewGoStepsContext().WithData(map[string]interface{}{
  "key1": "value1",
  "key2": "value2",
})

Step Function results

The step function must return a StepResult type, which contains the status of the step, message returned by the step function, and errors.

type StepResult struct {
  StepData    GoStepsCtxData `json:"stepData"`
  StepState   StepState      `json:"stepState"`
  StepMessage *string        `json:"stepMessage"`
  StepError   error     `json:"stepError,omitempty"`
}

GoSteps require a step function to return one of the following states.

StepStateComplete StepState = "StepStateComplete" // step completed successfully
StepStateFailed   StepState = "StepStateFailed"   // step failed to complete, without error
StepStateSkipped  StepState = "StepStateSkipped"  // step was skipped
StepStatePending  StepState = "StepStatePending"  // step is pending, should be retried
StepStateError    StepState = "StepStateError"    // step failed to complete, with error

Functions are defined to return the state of the step, and the message and error are optional. If the step is failed, the error should be returned.

gosteps.MarkStateComplete()
gosteps.MarkStateFailed()
gosteps.MarkStateSkipped()
gosteps.MarkStatePending()
gosteps.MarkStateError()

In addition to the state, the step function can also return a message and error.

// add data to the step result
gosteps.MarkStateComplete().WithData(map[string]interface{}{"key": value})

// add message to the step result
gosteps.MarkStateComplete().WithMessage("message")

// add error to the step result of type StepError
gosteps.MarkStateError().WithError(errors.New("error message"))

Conditional Branching

Some steps-chains might need to branch out conditionally, i.e run a different sets of steps based some condition. To achieve this, define the possible next Branches in the Branches field of the Step of type gosteps.Branches and pass the resolver function to the Resolver field.

stepWithBranches := gosteps.Step{
  Name:     "stepWithBranches",
  Function: Function,
  Branches: &gosteps.Branches{
    Resolver: func(ctx gosteps.GoStepsCtx) gosteps.BranchName {
      // do something
      return gosteps.BranchName("branch1")
    },
    Branches: []gosteps.Branch{
      {
        BranchName: "branch1",
        Steps:      gosteps.Steps{ /* steps */ },
      },
      {
        BranchName: "branch2",
        Steps:      gosteps.Steps{ /* steps */ },
      },
    },
  },
}

If a step function is defined for the step, the resolver runs after the function is executed. If no step function is defined the resolver function runs and returns the name of the branch to be executed next.

If the resolver function returns a branch name that is not defined in the branches, the execution will move on with execution of the main branch instead of any conditional branches.

Retrying a Step

Steps are retired if the StepState is not StepStateComplete or StepStateSkipped.

Steps are retried based for errors that are defined in StepOpts.ErrorsToRetry field. While steps are retried only for the stated errors, they can also be retried for any errors, irrespective of those mentioned in ErrorsToRetry, by setting StepOpts.RetryAllErrors as true.

The retry mechanism runs for StepOpts.MaxRunAttempts, that included the initial run. If a sleep duration is passed in the StepOpts.RetrySleep, then the step runner waits for that duration before executing the next retry of that step.

gosteps.StepOpts{
  ErrorsToRetry: []error{errors.New("error")},
  ErrorPatternsToRetry: []regexp.Regexp{*regexp.MustCompile("err*")},
  RetryAllErrors: false,
  MaxRunAttempts: 5,
  RetrySleep: 5 * time.Second,
}

Logging

GoSteps uses the [zerolog](https://github.com/rs/zerolog) package to enable logging within GoSteps. Initialize the logger using the gosteps.NewGoStepsLogger method, passing the output type and options.

// output : of type io.Writer, example os.Stdout, for more options refer to zerolog documentation: https://github.com/rs/zerolog?tab=readme-ov-file#multiple-log-output
output := zerolog.MultiLevelWriter(os.Stdout, runLogFile)

// opts   : of type *LoggerOpts, if nil, default options are used to enable step level logging, set StepLoggingEnabled to true
opts := LoggerOpts{ 
    StepLoggingEnabled : true,
}

logger := gosteps.NewGoStepsLogger(output, opts)

The logger can be passed to the GoStepsCtx.Use() method to enable logging.

ctx := gosteps.NewGoStepsContext()
ctx.Use(logger)

If StepLoggingEnabled is set to true then the step run is also logged with the step name, state, message, and error. In cases where logs are required within the step function, the logger can be accessed using the ctx.Log() method.

func(c gosteps.GoStepsCtx) gosteps.StepResult {
    c.Log("this is a message", gosteps.InfoLevel)
    return gosteps.MarkStateComplete()
}

Example

Sample code can be found in the example directory.

Documentation

Index

• Constants
• Variables
• func NewGoStepsLogger(out io.Writer, loggerOpts *LoggerOpts) goStepsLogger
• type Branch
  • func NewStepChain(steps Steps) *Branch
  • func NewStepsProcessor(steps Steps) *Branch
  • func (branch *Branch) Execute(c GoStepsContext)
  • func (branch *Branch) ToJson() (string, error)
• type BranchName
• type Branches
• type GoStepsContext
  • func NewGoStepsContext() GoStepsContext
• type GoStepsCtx
  • func (ctx GoStepsCtx) GetData(key string) interface{}
  • func (ctx GoStepsCtx) GetProgress(step StepName) StepProgress
  • func (c *GoStepsCtx) Log(message string, levels ...LogLevel)
  • func (ctx *GoStepsCtx) SetCurrentStep(step StepName) GoStepsCtx
  • func (ctx GoStepsCtx) SetData(key string, value interface{})
  • func (ctx *GoStepsCtx) SetProgress(step StepName, stepResult StepResult) GoStepsCtx
  • func (ctx *GoStepsCtx) Use(args ...interface{}) GoStepsContext
  • func (ctx GoStepsCtx) WithData(data map[string]interface{})
• type GoStepsCtxData
• type LogLevel
• type LoggerOpts
• type ResolverFn
• type Step
• type StepFn
• type StepName
• type StepOpts
• type StepProgress
• type StepResult
  • func MarkStateComplete() StepResult
  • func MarkStateError() StepResult
  • func MarkStateFailed() StepResult
  • func MarkStatePending() StepResult
  • func MarkStateSkipped() StepResult
  • func (sr StepResult) WithData(data GoStepsCtxData) StepResult
  • func (sr StepResult) WithError(stepErr error) StepResult
  • func (sr StepResult) WithMessage(message string) StepResult
• type StepRunProgress
• type StepState
• type Steps

Constants

const (
	// to avoid infinite runs due to the MaxAttempts not being set, we're keeping the default attempts to 100
	// if required, import and use the MaxMaxAttempts in the step.MaxAttempts field
	DefaultMaxAttempts = 100

	// the Max value is 9223372036854775807, which is not infinite but a huge number of attempts
	MaxMaxAttempts = math.MaxInt
)

Variables

var (

	// Log Level of GoSteps Logger implementation of zerolog.Level
	DebugLevel = LogLevel(zerolog.DebugLevel)
	InfoLevel  = LogLevel(zerolog.InfoLevel)
	WarnLevel  = LogLevel(zerolog.WarnLevel)
	ErrorLevel = LogLevel(zerolog.ErrorLevel)
)

Functions

func NewGoStepsLogger

func NewGoStepsLogger(out io.Writer, loggerOpts *LoggerOpts) goStepsLogger

NewGoStepsLogger returns a new instance of the GoStepsLogger

output: is of type io.Writer, example os.Stdout, for more options refer to zerolog documentation: https://github.com/rs/zerolog?tab=readme-ov-file#multiple-log-output

loggerOpts: is of type *LoggerOpts, if nil, default options are used to enable step level logging, set StepLoggingEnabled to true

Types

type Branch

type Branch struct {
	BranchName BranchName `json:"branchName"`
	Steps      Steps      `json:"steps"`
}

Branch type defines a unique step-chain, of the step-tree Branches can be used to define different steps to be executed based on a resolver function

func NewStepChain

func NewStepChain(steps Steps) *Branch

NewStepChain creates a new root branch of the step-chain Soon to be deprecated in favor of NewStepsProcessor

func NewStepsProcessor

func NewStepsProcessor(steps Steps) *Branch

NewStepsProcessor creates a new root branch of the step-chain

func (*Branch) Execute

func (branch *Branch) Execute(c GoStepsContext)

Execute a branch with the context provided

func (*Branch) ToJson

func (branch *Branch) ToJson() (string, error)

ToJson converts the step-tree to JSON-string

type BranchName

type BranchName string

BranchName type defined the name of the branch

type Branches

type Branches struct {
	Branches []Branch   `json:"branches"`
	Resolver ResolverFn `json:"-"`
}

Branches type defines a list of branches with a resolver function to determine the branch to execute

type GoStepsContext

type GoStepsContext interface {
	Use(args ...interface{}) GoStepsContext
	Log(message string, levels ...LogLevel)
	SetData(key string, value interface{})
	GetData(key string) interface{}
	WithData(data map[string]interface{})
	SetProgress(step StepName, stepResult StepResult) GoStepsCtx
	SetCurrentStep(step StepName) GoStepsCtx
	// contains filtered or unexported methods
}

GoStepsContext interface defines the methods for the context

func NewGoStepsContext

func NewGoStepsContext() GoStepsContext

GoStepsCtx type defines the context for the step-chain

type GoStepsCtx

type GoStepsCtx struct {
	// contains filtered or unexported fields
}

GoStepsCtx type defines the context for the step-chain

func (GoStepsCtx) GetData

func (ctx GoStepsCtx) GetData(key string) interface{}

GetData gets the data from the context

func (GoStepsCtx) GetProgress

func (ctx GoStepsCtx) GetProgress(step StepName) StepProgress

GetProgress gets the progress of the step

func (*GoStepsCtx) Log

func (c *GoStepsCtx) Log(message string, levels ...LogLevel)

Log logs the message with the step name and the log level, if provided. The log level is the first argument in the args.

func (*GoStepsCtx) SetCurrentStep

func (ctx *GoStepsCtx) SetCurrentStep(step StepName) GoStepsCtx

SetCurrentStep sets the current step

func (GoStepsCtx) SetData

func (ctx GoStepsCtx) SetData(key string, value interface{})

SetData sets the data in the context

func (*GoStepsCtx) SetProgress

func (ctx *GoStepsCtx) SetProgress(step StepName, stepResult StepResult) GoStepsCtx

SetProgress sets the progress of the step

func (*GoStepsCtx) Use

func (ctx *GoStepsCtx) Use(args ...interface{}) GoStepsContext

Handles adding handlers to the context

func (GoStepsCtx) WithData

func (ctx GoStepsCtx) WithData(data map[string]interface{})

WithData sets the data in the context

type GoStepsCtxData

type GoStepsCtxData map[string]interface{}

GoStepsCtxData type defines the data stored in the context

type LogLevel

type LogLevel zerolog.Level

type LoggerOpts

type LoggerOpts struct {
	StepLoggingEnabled bool
}

type ResolverFn

type ResolverFn func(ctx GoStepsCtx) BranchName

ResolverFn defines the Resolver Function to determine the branch to execute

type Step

type Step struct {
	Name     StepName               `json:"name"`
	Function StepFn                 `json:"-"`
	StepOpts StepOpts               `json:"stepConfig"`
	Branches *Branches              `json:"branches"`
	StepArgs map[string]interface{} `json:"stepArgs"`
	// contains filtered or unexported fields
}

Step type defines a step with all configurations for the step

type StepFn

type StepFn func(ctx GoStepsCtx) StepResult

StepFn defines the Step's Function

type StepName

type StepName string

StepName type defined the name of the step

type StepOpts

type StepOpts struct {
	ErrorsToRetry        []error         `json:"errorsToRetry"`
	ErrorPatternsToRetry []regexp.Regexp `json:"errorPatternsToRetry"`
	RetryAllErrors       bool            `json:"retryAllErrors"`
	MaxRunAttempts       int             `json:"maxAttempts"`
	RetrySleep           time.Duration   `json:"retrySleep"`
}

StepOpts type defines the configuration for the step

type StepProgress

type StepProgress struct {
	StepName   StepName   `json:"stepName"`
	StepResult StepResult `json:"stepResult"`
}

StepProgress type defines the progress of the step

type StepResult

type StepResult struct {
	StepData    GoStepsCtxData `json:"stepData"`            // stores the data from a step, if any
	StepState   StepState      `json:"stepState"`           // state of the step
	StepMessage *string        `json:"stepMessage"`         // message from the step execution, if any
	StepError   error          `json:"stepError,omitempty"` // error from the step execution, if any
}

StepResult type defines the result of the step

func MarkStateComplete

func MarkStateComplete() StepResult

MarkStateComplete marks the state of the step as complete

func MarkStateError

func MarkStateError() StepResult

MarkStateError marks the state of the step as error

func MarkStateFailed

func MarkStateFailed() StepResult

MarkStateFailed marks the state of the step as failed

func MarkStatePending

func MarkStatePending() StepResult

MarkStatePending marks the state of the step as pending

func MarkStateSkipped

func MarkStateSkipped() StepResult

MarkStateSkipped marks the state of the step as skipped

func (StepResult) WithData

func (sr StepResult) WithData(data GoStepsCtxData) StepResult

WithData sets the data for the step

func (StepResult) WithError

func (sr StepResult) WithError(stepErr error) StepResult

WithStepError sets the GoSteps - StepError for the step

func (StepResult) WithMessage

func (sr StepResult) WithMessage(message string) StepResult

WithMessage sets the message for the step

type StepRunProgress

type StepRunProgress struct {
	// contains filtered or unexported fields
}

stepRunProgress type defines the progress of the step it contains the run/execution count of each step

type StepState

type StepState string

StepState type defines the state of the step after execution

const (
	StepStateComplete StepState = "StepStateComplete" // step completed successfully             [non-retriable]
	StepStateFailed   StepState = "StepStateFailed"   // step failed to complete, without error  [non-retriable]
	StepStateSkipped  StepState = "StepStateSkipped"  // step was skipped                        [non-retriable]
	StepStatePending  StepState = "StepStatePending"  // step is pending, should be retried      [retriable]
	StepStateError    StepState = "StepStateError"    // step failed to complete, with error     [retriable]
)

type Steps

type Steps []Step

Steps type defines a list of steps