verify

Documentation

Overview

Package verify decides whether an exercise is solved.

Index

• func Render(w io.Writer, ex exercise.Exercise, r Result)
• func RenderChecklist(w io.Writer, ex exercise.Exercise)
• func RunSolutionTimes(ctx context.Context, root string, ex exercise.Exercise, n int, ...) (runs [][]string, exitCode int, err error)
• type Result
  • func Verify(ctx context.Context, root string, ex exercise.Exercise) Result
  • func VerifySolution(ctx context.Context, root string, ex exercise.Exercise) Result
• type RunLog
• type Stage

Functions

func Render

func Render(w io.Writer, ex exercise.Exercise, r Result)

Render writes a failure report. The failure is the teaching moment, so it says which stage broke and shows the diff aligned, rather than dumping two blobs and leaving the learner to spot the difference.

func RenderChecklist

func RenderChecklist(w io.Writer, ex exercise.Exercise)

RenderChecklist prints a ModeManual exercise's checks.

func RunSolutionTimes

func RunSolutionTimes(ctx context.Context, root string, ex exercise.Exercise, n int, envPerRun [][]string) (runs [][]string, exitCode int, err error)

RunSolutionTimes builds an exercise's solution once and runs it n times, returning what each run printed. `babygo capture` uses it to derive expected_output from real behavior rather than from upstream's prose transcript, which is in several places stale or elided.

n matters. Classifying determinism from two runs is unsound when the nondeterminism is coarse: range-over-built-in-types iterates a two-key map, and Go randomizes map order, so two runs agree half the time — it was confidently classified "exact" on a coin flip. Ten runs cut that to about one in five hundred.

Types

type Result

type Result struct {
	Passed bool
	Stage  Stage // the stage that failed; meaningless when Passed

	// CompileErr is the compiler's message, when Stage is StageCompile.
	CompileErr string

	// Expected and Got are the compared outputs, when Stage is StageOutput.
	// They are the first invocation that disagreed, not all of them.
	Expected []string
	Got      []string

	// Invocation identifies which run failed, for exercises with several.
	Invocation string

	// TestOutput is `go test` output, when the mode is ModeTest.
	TestOutput string

	// Runs holds every invocation's output when Passed. A correct answer is
	// not a reason to hide what the program printed — seeing the output is
	// most of the point of running it.
	Runs []RunLog

	// Err is an infrastructure failure — a missing file, a broken timeout —
	// as opposed to the learner's code being wrong.
	Err error
}

Result is the outcome of verifying one exercise.

func Verify

func Verify(ctx context.Context, root string, ex exercise.Exercise) Result

Verify compiles and runs an exercise and checks its output against the mode declared in its meta.json. dir is the repo root.

Compilation is a separate `go build` rather than `go run` so that a compiler error is distinguishable from a program that ran and printed the wrong thing. With `go run` both arrive on stderr and the runner cannot tell the learner which of the two happened — which is the single most useful thing it can say.

func VerifySolution

func VerifySolution(ctx context.Context, root string, ex exercise.Exercise) Result

VerifySolution runs the same checks against solutions/ instead of exercises/. `make verify` uses it to prove every expected_output is actually achievable — a mis-parsed expected file would otherwise make an exercise unsolvable, and nobody would find out until a learner lost an hour to it.

type RunLog

type RunLog struct {
	Invocation string
	Lines      []string
}

RunLog is one invocation's captured output.

type Stage

type Stage string

Stage is how far an exercise got before failing.

const (
	StageCompile Stage = "compile"
	StageRun     Stage = "run"
	StageOutput  Stage = "output"
	StageManual  Stage = "manual"
)