evalrunner

Documentation

Overview

Package evalrunner runs Braintrust evals under the `bt` CLI.

A program built with this package is not a server. bt owns the HTTP port, authentication and CORS; it spawns this binary once per request, passes the request in environment variables, and reads results back over a unix socket. The process runs one eval and exits.

A minimal runner:

func main() {
    r := evalrunner.New()

    evalrunner.RegisterEval(r, classify)

    evalrunner.Main(r)
}

Then point bt at the package directory:

bt eval --dev --language go ./cmd/evals

Decoding the request and dispatching to the right eval happens inside Main; registered evals are looked up by their eval.Eval Name.

Index

• func Main(r *Runner)
• func RegisterEval[I, R any](r *Runner, ev *eval.Eval[I, R])
• func Run(ctx context.Context, r *Runner) error
• type EvalData
• type EvalRequest
• type Mode
• type Option
  • func WithLogger(l logger.Logger) Option
  • func WithTracerProvider(tp *sdktrace.TracerProvider) Option
• type ParentInfo
• type Runner
  • func New(opts ...Option) *Runner
  • func (r *Runner) Mode() Mode

Functions

func Main

func Main(r *Runner)

Main runs the runner and exits the process. It never returns.

It exists so the common case is one line. Programs that need to do their own cleanup should call Run instead and handle the error themselves.

func RegisterEval

func RegisterEval[I, R any](r *Runner, ev *eval.Eval[I, R])

RegisterEval adds an eval definition to the runner. The type parameters I and R are the input and result types of the evaluation. Go does not allow generic methods on non-generic types, so this is a package-level function.

The eval's Name is its lookup key: it is what the Braintrust playground shows and what bt sends back to select this eval. Registering two evals under the same name replaces the first.

Example:

classify := &eval.Eval[string, string]{
    Name:    "classify",
    Task:    eval.T(classifyTask),
    Scorers: []eval.Scorer[string, string]{scorer},
}
evalrunner.RegisterEval(r, classify)

func Run

func Run(ctx context.Context, r *Runner) error

Run dispatches on the environment bt provided and blocks until finished.

os.Args is deliberately ignored: bt constructs argv for its own runner scripts and splices arguments in that mean nothing to us.

Types

type EvalData

type EvalData struct {
	// Data is an inline array of test cases.
	Data json.RawMessage `json:"data,omitempty"`

	// DatasetID loads a dataset by ID.
	DatasetID string `json:"dataset_id,omitempty"`

	// DatasetName loads a dataset by name, scoped by ProjectID or ProjectName.
	DatasetName string `json:"dataset_name,omitempty"`

	// ProjectID scopes DatasetName lookups. This is what bt sends.
	ProjectID string `json:"project_id,omitempty"`

	// ProjectName scopes DatasetName lookups (optional).
	ProjectName string `json:"project_name,omitempty"`
}

EvalData specifies where evaluation data comes from.

The playground sends a dataset_id, but bt resolves that into {project_id, dataset_name} before spawning us and discards every other key, so ProjectID+DatasetName is the shape that actually arrives in practice. See bt/src/eval.rs:1256 (resolve_dataset_ref_for_eval_request).

type EvalRequest

type EvalRequest struct {
	// Name is the registered evaluator name (required).
	Name string `json:"name"`

	// Data specifies the evaluation dataset (required).
	Data EvalData `json:"data"`

	// ExperimentName overrides the experiment name (optional).
	ExperimentName string `json:"experiment_name,omitempty"`

	// ProjectID overrides the project ID (optional).
	ProjectID string `json:"project_id,omitempty"`

	// Parameters holds the resolved parameter values selected in the playground
	// (a flat name->value map). Merged over declared defaults and surfaced to the
	// task via eval.TaskHooks.Parameters.
	Parameters map[string]any `json:"parameters,omitempty"`

	// Parent specifies the parent span for tracing (optional).
	Parent *ParentInfo `json:"parent,omitempty"`
}

EvalRequest is the request body for POST /eval.

type Mode

type Mode string

Mode is what bt is asking this process to do. It is derived entirely from the environment, never from os.Args -- bt splices its own arguments in when a runner binary is used, so argv is meaningless to us.

const (
	// ModeList means bt wants the manifest of registered evals, printed as JSON
	// on stdout. No authentication and no API calls are needed.
	ModeList Mode = "list"

	// ModeEval means bt wants one specific eval run, described by
	// BT_EVAL_DEV_REQUEST_JSON, with results streamed back over the socket.
	ModeEval Mode = "eval"

	// ModeBatch is `bt eval <target>` without the dev server: run every
	// registered eval, honouring the CLI's filter and output flags.
	ModeBatch Mode = "batch"

	// ModeInspect means nothing spawned us -- somebody ran the binary directly.
	// Print what is registered and exit without touching Braintrust.
	ModeInspect Mode = "inspect"

	// ModeUnknown means BT_EVAL_DEV_MODE held a value we do not recognise.
	ModeUnknown Mode = "unknown"
)

type Option

type Option func(*Runner)

Option configures a Runner.

func WithLogger

func WithLogger(l logger.Logger) Option

WithLogger sets a custom logger. Logs must go to stderr: bt reads the eval manifest from stdout.

func WithTracerProvider

func WithTracerProvider(tp *sdktrace.TracerProvider) Option

WithTracerProvider sets a custom OpenTelemetry TracerProvider.

Supply one when instrumented code (LLM clients, custom spans) should appear in the same trace as eval spans. When nil, a provider is created per run and shut down before the process exits.

type ParentInfo

type ParentInfo struct {
	ObjectType      string          `json:"object_type,omitempty"`
	ObjectID        string          `json:"object_id,omitempty"`
	PropagatedEvent json.RawMessage `json:"propagated_event,omitempty"`
}

ParentInfo specifies parent span context for tracing.

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner holds the evals a program exposes to bt.

Registration happens once, before Run, from a single goroutine, so the evaluator map needs no locking.

func New

func New(opts ...Option) *Runner

New creates a runner.

func (*Runner) Mode

func (r *Runner) Mode() Mode

Mode reports what bt is asking this process to do. Use it to skip expensive setup when bt only wants the list of evals:

if r.Mode() != evalrunner.ModeList {
    warmCaches()
}