benchmark-workers

README

benchmark-workers

Pre-written workflows and activities useful for benchmarking Temporal.

This worker can be used alongside Maru or other benchmarking tools to mimic different workloads.

Also included is a simple workflow runner which will keep a configurable number of workflow executions running concurrently to provide load for testing, starting a new execution each time one completes.

Usage

Worker

The worker is available as docker image for use in Docker or Kubernetes setups.

You can pull the latest image from: ghcr.io/temporalio/benchmark-workers:main.

In future we will provide releases with appropriate image tags to make benchmarks more easily repeatable.

The worker can be configured via environment variables. Currently only a small number of options are available, please let us know if there is a particular option you would like to be exposed by filing an issue.

The table below lists the environment variables available and the relevant Temporal Go SDK options they relate to (the worker is currently written using the Temporal Go SDK).

Environment Variable	Relevant Client or Worker option	Description
TEMPORAL_GRPC_ENDPOINT	ClientOptions.HostPort	The Temporal Frontend GRPC endpoint
TEMPORAL_TLS_KEY	ClientOptions.ConnectionOptions.TLS	Path to TLS Key file
TEMPORAL_TLS_CERT	ClientOptions.ConnectionOptions.TLS	Path to TLS Cert file
TEMPORAL_TLS_CA	ClientOptions.ConnectionOptions.TLS	Path to TLS CA Cert file
TEMPORAL_NAMESPACE	ClientOptions.Namespace	The Temporal Namespace
TEMPORAL_TASK_QUEUE	TaskQueue	The Temporal Task Queue
TEMPORAL_WORKFLOW_TASK_POLLERS	WorkerOptions.MaxConcurrentWorkflowTaskPollers	Number of workflow task pollers
TEMPORAL_ACTIVITY_TASK_POLLERS	WorkerOptions.MaxConcurrentActivityTaskPollers	Number of activity task pollers
PROMETHEUS_ENDPOINT	n/a	The address to serve prometheus metrics on

To run the worker in a Kubernetes cluster you could use:

kubectl run benchmark-worker --image ghcr.io/temporalio/benchmark-workers:main \
    --image-pull-policy Always \
    --env "TEMPORAL_GRPC_ENDPOINT=temporal-frontend.temporal:7233" \
    --env "TEMPORAL_NAMESPACE=default" \
    --env "TEMPORAL_TASK_QUEUE=benchmark" \
    --env "TEMPORAL_WORKFLOW_TASK_POLLERS=16" \
    --env "TEMPORAL_ACTIVITY_TASK_POLLERS=8"

However, we suggest you use a deployment for workers rather than kubectl run so that you can collect metrics via prometheus. We provide an example deployment spec for you to customize to your requirements. Once you have edited the environment variables in the deployment.yaml you can create the deployment with kubectl apply -f ./deployment.yaml.

You can then use the benchmark workflows with your benchmark tool. To test with tctl you could run:

tctl workflow start --taskqueue benchmark --workflow_type ExecuteActivity --execution_timeout 60 -i '{"Count":1,"Activity":"Sleep","Input":{"SleepTimeInSeconds":3}}'

This will run the ExecuteActivity workflow, described below.

Runner

The runner is available as docker image for use in Docker or Kubernetes setups.

You can pull the latest image from: ghcr.io/temporalio/benchmark-workers:main.

The runner can be configured via environment variables and command line arguments. Currently only a small number of options are available, please let us know if there is a particular option you would like to be exposed by filing an issue.

The table below lists the environment variables available and the relevant Temporal Go SDK options they relate to (the runner is currently written using the Temporal Go SDK).

Environment Variable	Relevant Client or Worker option	Description
TEMPORAL_GRPC_ENDPOINT	ClientOptions.HostPort	The Temporal Frontend GRPC endpoint
TEMPORAL_TLS_KEY	ClientOptions.ConnectionOptions.TLS.Certificates	Path to TLS Key file
TEMPORAL_TLS_CERT	ClientOptions.ConnectionOptions.TLS.Certificates	Path to TLS Cert file
TEMPORAL_TLS_CA	ClientOptions.ConnectionOptions.TLS	Path to TLS CA Cert file
PROMETHEUS_ENDPOINT	n/a	The address to serve prometheus metrics on

The runner is also configured via command line options:

Usage: runner [flags] [workflow input] ...
  -c int
    	concurrent workflows (default 10)
  -n string
    	namespace (default "default")
  -t string
    	workflow type
  -tq string
    	task queue (default "benchmark")
  -w	wait for workflows to complete (default true)

To use the runner in a Kubernetes cluster you could use:

kubectl run benchmark-runner --image ghcr.io/temporalio/benchmark-workers:main \
    --image-pull-policy Always \
    --env "TEMPORAL_GRPC_ENDPOINT=temporal-frontend.temporal:7233" \
    --command -- runner -t ExecuteActivity '{ "Count": 3, "Activity": "Echo", "Input": { "Message": "test" } }'

Workflows

The worker provides the following workflows for you to use during benchmarking:

ExecuteActivity

ExecuteActivity({ Count: int, Activity: string, Input: interface{} })

This workflow takes a count, an activity name and an activity input. The activity Activity will be run Count times with the given input. If the activity returns an error the workflow will fail with that error.

Activities

The worker provides the following activities for you to use during benchmarking:

Sleep

Sleep({ SleepTimeInSeconds: int })

This activity sleeps for the given number of seconds. It never returns an error. This can be used to simulate activities which take a while to complete.

Echo

Echo({ Message: string }) result

This activity simply returns the message as it's result. This can be used for stress testing polling with activities that return instantly.