Documentation
¶
Overview ¶
Command eval shows how to use the statistical evaluation harness. It builds an agent, defines labeled cases and metrics, runs each case several times, and prints a pass-rate report.
The model here is a stub sentiment classifier that is deliberately flaky (it gets one in five calls wrong) so the report shows a rate below 100%, which is the point: the model is stochastic, so eval reports a distribution, not a verdict. To evaluate a real agent, replace the stub with anthropic.New / openai.New / gemini.New and keep everything else the same.
Remember the boundary (see docs/testing/testing.md): eval measures the model statistically; the provable guarantees (governed convergence, cryptographic audit) live in the govern and audit packages. A pass rate is a signal, not a proof.
Source Files