conformance

Documentation

Overview

Package conformance reads the conformance corpus and compares a runner's answer against what an entry expects.

The corpus is testdata/conformance/, its format is docs/conformance/SPEC.md, and this package is the reading of that specification which this repository runs. An entry is the tuple the spec describes: the layout and the copybooks it names, the IR they resolve to, the bytes of a file laid out that way, and the values those bytes decode to (#66). Which entries there are and what each was derived from stay with the corpus, in that directory's README.md.

Why a corpus exists at all

cpybkc is a code generator and nothing else, so there is no command of its own that reads a data file. What decodes bytes is the code a generator emitted, in whatever language it emitted it, and nothing about the IR obliges two generators to agree about a byte they were both handed. docs/ir/SPEC.md names that hazard twice — a position sum every consumer runs and every consumer may get wrong on its own, and two writers free to choose different bytes for one record — and both times it names this corpus as what catches it. A shared set of files with the answer written down is the only mechanism that can: it is what protobuf's conformance suite is, and for the same reason.

The two halves, and which one is here

An entry states two independent things, and they are checked by different callers.

• The layout and its copybooks resolve to the entry's IR. That is a claim about a *producer*, and the pipeline that would check it is the CLI's (#148); the descriptor an entry carries is hand-authored, so it is an oracle rather than a recording of what resolve happens to do today.
• The IR, handed to a generator, produces code that decodes the entry's bytes into the entry's values — and writes those records back into a file that decodes to the entry's values again. That is a claim about a *consumer*, in both directions, it is what github.com/Zaba505/cpybkc/internal/conformance/engine asks an adapter for — including the one this repository ships for its own generator, github.com/Zaba505/cpybkc/internal/conformance/goadapter — and it is what makes the corpus useful to a third party who has neither this repository's resolver nor its language. Answer carries both answers and says why the writing direction is checked by reading rather than by comparing bytes (#68).

Both halves read the same entry, which is the point of carrying the IR in the tuple rather than deriving it: a generator author who disagrees with cpybkc about what a layout means and a generator author who disagrees about what a descriptor means have different bugs, and an entry that carried only three of the four members could not tell them apart.

What Load checks, and what it declines to

Loading an entry parses every member and holds it to what the format requires: entry.json carries a description, a source, at most a Status and nothing else, ir.json is a descriptor that passes github.com/Zaba505/cpybkc/internal/assemble.Validate and is written in the canonical rendering github.com/Zaba505/cpybkc/internal/emit.MarshalJSON produces, values.json names records the descriptor carries and spells every scalar the way the value language spells one, and the directory holds no file the format has no place for.

A spelling is checked here rather than left to the comparison because "012" and "12" are one value written down twice — an author's typo, not something two implementations can disagree about — and a generator appearing to disagree about a number sends its author to their decoder (#196).

It does not check that the values are the *right* values for the bytes. Nothing here decodes anything: that is what a runner does, and a loader that decoded a record would be a second implementation of the thing the corpus exists to test. A stray file is refused for the reason the project manifest refuses an unknown field — a file somebody added to an entry directory expecting it to be read is worse than one they are told about.

Index

• Constants
• func CheckDigest(dir string) (string, bool, error)
• func Compare(want, got *Values) error
• func CompareAnswer(want *Values, got *Answer) error
• func CorpusPath(root string) string
• func Digest(dir string) (string, error)
• func DigestFS(corpus fs.FS) (string, error)
• func DigestPath(dir string) string
• func FormatDigest(digest string) []byte
• func FormatFloat(f float64) string
• func ReadDigest(path string) (string, error)
• func WriteScalar(value reflect.Value) (string, error)
• func WriteValue(nodes map[uint64]*irpb.Node, node *irpb.Node, value reflect.Value) (any, error)
• type Answer
  • func ParseAnswer(b []byte) (*Answer, error)
• type Entry
  • func Load(dir string) ([]*Entry, error)
  • func LoadEntry(dir string) (*Entry, error)
  • func (e *Entry) IsNormative() bool
  • func (e *Entry) IsProvisional() bool
• type EntryError
  • func (e *EntryError) Error() string
  • func (e *EntryError) Unwrap() error
• type MismatchError
  • func (e *MismatchError) Error() string
  • func (e *MismatchError) Unwrap() error
• type PathError
  • func (e *PathError) Error() string
  • func (e *PathError) Unwrap() error
• type Record
• type RunError
  • func (e *RunError) Error() string
  • func (e *RunError) Unwrap() error
• type Status
• type Values
  • func ParseValues(b []byte) (*Values, error)

Constants

const (
	// MetadataName carries what an entry is and where it came from.
	MetadataName = "entry.json"

	// LayoutName is the layout the entry's file is laid out by.
	LayoutName = "layout.sexpr"

	// DescriptorName is the IR the layout and its copybooks resolve to, in the
	// canonical JSON rendering --emit-ir-format json writes.
	DescriptorName = "ir.json"

	// InputName is the bytes of one file laid out that way.
	InputName = "input.bin"

	// ValuesName is what those bytes decode to.
	ValuesName = "values.json"

	// OffsetsName is a member of an entry that is reserved and not specified:
	// an entry may carry one, nothing reads it, and no entry does (#194).
	//
	// The name is admitted here rather than left to be added later because
	// adding a member to a published format is a coordinated change across
	// every runner that holds an entry's listing to the set of names it knows,
	// and reserving one before the format is public costs nothing. What would
	// go in it is a per-field offset table — the oracle a generator that never
	// reads bytes could be held to, since it runs the same position sum a
	// decoder runs (#193) — and whether a corpus entry is where that belongs is
	// open, which is why the name is reserved and the content is not specified.
	OffsetsName = "offsets.json"

	// CopybookExt is the extension every copybook of an entry carries. The
	// layout names them; what this decides is only which files in the directory
	// are allowed to be there.
	CopybookExt = ".cpy"
)

The names an entry's members are written under. They are fixed rather than declared in entry.json because a tuple whose members could be anywhere is a tuple every reader has to be told about, and a third-party runner reading the corpus should be able to open a file without parsing something first.

const CorpusFile = "testdata/conformance"

CorpusFile is where the corpus lives, relative to the repository root, in the slash-separated spelling a repository path is written in.

It is a constant here for the reason github.com/Zaba505/cpybkc/internal/layoutschema.SchemaFile is one: the tests, the runners and anything else that means "the corpus" have to mean one directory, and a path written out in three places is a path two of them get wrong.

const DigestExt = ".sha256"

DigestExt is appended to a corpus directory's path to name the file its digest is published in: the corpus at `corpus/` is checked against `corpus.sha256`.

Beside the corpus rather than inside it, and that is not a filing preference. A digest of a directory cannot live in that directory: it would have to cover itself. Putting it inside would mean naming one member every reader has to be told to leave out — which is a change to the corpus format (docs/conformance/SPEC.md, *An entry*) and a change Load would have to carry, for the sake of a file that is not part of the corpus but a statement about it.

const PublishedCorpusDir = "corpus"

PublishedCorpusDir is where the corpus sits inside cpybkc-conformance.tar.gz, the archive a release attaches (#202).

It is a second path because a release asset is not a checkout: an adopter unpacks the archive and works in it, and `testdata/` there would name a convention of a Go repository they do not have. It is a constant here, beside CorpusFile, for exactly the reason that one is — the tool that writes the archive and the command that reads one have to mean the same directory, and they are two programs.

Functions

func CheckDigest

func CheckDigest(dir string) (string, bool, error)

CheckDigest digests the corpus at dir and holds it against the digest published beside it, at DigestPath.

The digest it computed comes back either way, so that a caller can report which corpus it ran against whether or not there was anything to check it against.

checked is false where no digest file is there at all, and that is not an error. The corpus in this repository's own tree has none and cannot: the digest is a function of the corpus, so a committed copy would be a second statement of it for an entry to be added without — the same reason CONTRIBUTING.md, *The release artifacts*, gives for ir.binpb not being committed. A caller that ran against an unchecked corpus should say so rather than report a bare pass.

func Compare

func Compare(want, got *Values) error

Compare reports every way in which got differs from want.

The comparison is structural and knows no COBOL: both sides have already been written in the corpus's own value language, where a number is a decimal string and a run of bytes is base64, precisely so that comparing them needs neither the descriptor nor a decoder. What that buys is that a runner for a language this repository has never seen is compared by exactly this function.

Every difference is reported rather than the first, each naming the path through the record it is at, because a generator that gets one axis of the encoding wrong gets every field along that axis wrong at once and a report naming one of them sends its author looking for a single-field bug.

func CompareAnswer

func CompareAnswer(want *Values, got *Answer) error

CompareAnswer holds a runner's answer against what an entry expects, in both directions.

One entry, one set of values, and both directions are held to it: the records the generated reader made of the entry's bytes, and the records read back out of the file the generated writer made of those records. The second comparison is against the entry rather than against the first answer on purpose — a reader and a writer that are wrong the same way agree with each other, and only the entry knows what the file holds.

func CorpusPath

func CorpusPath(root string) string

CorpusPath is CorpusFile under root, in the host's own path spelling.

func Digest

func Digest(dir string) (string, error)

Digest reduces the corpus at dir to one SHA-256, written lowercase hexadecimal.

The rule, so that a third party can recompute it

Every regular file under dir contributes, in ascending order of its slash-separated path relative to dir. For each one the hash is fed

the path, a zero byte, the length in decimal, a zero byte, then the bytes

and the digest is the sum. Directories contribute nothing of their own, so an empty one is not part of a corpus; anything that is not a regular file is an error rather than something to skip, because a symbolic link in a published corpus is a file whose content depends on where it was unpacked.

The length is in there because a corpus holds arbitrary bytes — `input.bin` is bytes by definition — so the contents cannot be their own delimiter. Without it two different corpora whose concatenations happened to coincide would agree, which is the one property a digest exists to deny. The path goes in for the same reason: a file that moved between entries is a different corpus, and a digest over contents alone would not say so.

Nothing about a *file* other than its path and its bytes contributes. A corpus is text and bytes an author wrote down, so a mode, an owner or a modification time here would make the digest a function of the unpacking as well as of the corpus — and the same archive unpacked twice would fail its own check.

func DigestFS

func DigestFS(corpus fs.FS) (string, error)

DigestFS is Digest over an io/fs.FS.

It is exported for the producer's sake rather than for testing. The tool that writes cpybkc-conformance.tar.gz has to digest exactly the tree it is archiving, and a second walk of the same directory is a second read that a concurrent edit can make disagree with the first — which would publish a digest for a corpus the archive does not hold. One filesystem, archived and digested, is what makes the two the same corpus by construction.

func DigestPath

func DigestPath(dir string) string

DigestPath is where the digest of the corpus at dir is published.

A corpus named `.` — which is what `--corpus .` from inside an unpacked corpus means, and a plausible thing for somebody to type — has no name for a file to sit beside, and appending to it produces `..sha256`: a path nobody can have created, so the digest reads as absent and the run proceeds unchecked. Resolving against the working directory is what gives it one. It is done only in that case, because a relative path that already has a name reads far better in a diagnostic than the absolute one it would become.

func FormatDigest

func FormatDigest(digest string) []byte

FormatDigest is what a digest file holds: the digest and a newline, and nothing else.

Deliberately not sha256sum's `<digest> <name>` line. That format names a file, this digest covers a directory, and a line that looked like sha256sum's would invite `sha256sum -c` — which would check one file that is not there and report success for a corpus it never read.

func FormatFloat

func FormatFloat(f float64) string

FormatFloat writes a COMP-1 or COMP-2 value in the form the corpus's value language states: one of three sentinels, or the exact value in hexadecimal significand notation. See docs/conformance/SPEC.md, "A float is written exactly, and never as a JSON number", which this function is the Go reading of (#194, #195).

It is never a JSON number. Four things break under one and each is why this form exists: NaN and the infinities are not JSON numbers and most writers refuse them outright, so an honest generator decoding an IEEE NaN would make a harness that could not write a document at all; a negative zero and a positive one compare equal under IEEE equality, so a generator that lost the sign of a zero would pass; an authored decimal is not a COMP-1 value, so an entry's author would have to compute the double an implementation must print; and a decimal detour is lossy in a way this one is not, since the hexadecimal form is a transcription of the significand and the exponent rather than a conversion between radices.

What that last argument does not claim is full HFP precision. HFP long carries a 56-bit fraction against a binary64's 53, so a value using the last three is already rounded by the decoder that produced it — this function takes a float64, and no spelling recovers a bit that was lost before it was called. Stating those values would need a wider type here and a driver that handed over something other than a float64; nothing generates them today, and the day something does is the day to widen it.

One value has exactly one spelling, which is what lets the comparison be string equality. strconv.FormatFloat with 'x' writes the value exactly but not canonically — it pads the exponent to two digits, so 1 comes back as 0x1p+00 — so its output is normalized here rather than returned. Every other language's hexadecimal float formatter differs from the canonical form in a way of its own, which the spec says at length; each implementation normalizes its own.

A float32 is passed through a float64, which is exact: every binary32 value is a binary64 value, and the hexadecimal form of the widened value is the hexadecimal form of the original.

func ReadDigest

func ReadDigest(path string) (string, error)

ReadDigest reads a digest file and returns the digest it holds.

func WriteScalar

func WriteScalar(value reflect.Value) (string, error)

WriteScalar is what an elementary item holds, written the way the value language writes it: characters as text with every trailing space removed, a number as its decimal digits, a float in hexadecimal significand notation, and a run of bytes as base64.

Every one of those is a JSON string. A float in particular is never a JSON number: NaN and the infinities cannot be marshalled as one, so an honest generator decoding an IEEE NaN would make a driver fail to write a document at all — which the corpus format defines as the harness breaking, the one outcome that is not a conformance failure.

Which of the four an item takes is a function of its usage and category, and it is not read here: the generated code has already applied it, in the Go type it gave the field. docs/conformance/SPEC.md's "Which form a value takes is decided by the descriptor" is the mapping, and cmd/cpybkc-gen-go's README.md is the Go type each row of it lands on.

The trailing spaces are removed here rather than relied on to have been removed upstream. The rule is the format's requirement on *a writer of a values document*, and this is one; a decoder that trims is a decoder that happens to agree with it, and a value language whose rule is enforced only by a dependency is a rule this repository cannot be held to. Trimming an already trimmed value costs nothing and changes no answer the corpus holds today.

It does change what a corpus run *measures*, and in the direction of measuring less: a generator whose accessor left the padding on an alphanumeric item used to fail every entry that carries one, and now has it trimmed away here instead. That is deliberate. What the corpus compares is two readings of one descriptor written in one language, and the padding is not part of either — docs/conformance/SPEC.md says at length that it is a width the value language does not carry, so an entry that failed on it was reporting a disagreement about the *document* rather than about the record. What an accessor returns is cmd/cpybkc-gen-go's business and its own tests', and a corpus entry is the wrong instrument for it: no entry could say which of the two ends of the pipeline had kept the spaces.

The return is a string rather than an any because every one of the four forms is one — every scalar of the value language is a JSON string, which is what makes a comparison of two answers string equality. WriteValue is the looser type, because a node can be an object or an array as well.

func WriteValue

func WriteValue(nodes map[uint64]*irpb.Node, node *irpb.Node, value reflect.Value) (any, error)

WriteValue is what one node of a descriptor holds, written in the value language: an object for a group, an array where the node repeats, and a string for an elementary item, in the spelling docs/conformance/SPEC.md's "The value language" states.

The value is a reflect.Value over whatever the generated code decoded the record into, walked beside the descriptor rather than named field by field, which is what lets one writer serve every entry. What that requires of the decoded value is only what the Go generator's own record types provide: a group is a struct whose exported fields stand for its members in order, a repeating node is an array or a slice, a variant's arms are one pointer each of which exactly one is non-nil, and an elementary item is a string, an integer, a float or a run of bytes.

It lives here, beside the reader of the same language, rather than in the codec program github.com/Zaba505/cpybkc/internal/conformance/goadapter writes, for the reason docs/conformance/GRAMMAR.md exists at all: a values document that is written wrongly presents as a generator that decoded wrongly, which is the most expensive way there is to learn about a formatting mistake. Code inside a template is checked by compiling a scratch program per corpus entry and by nothing else, so the one part of it that is about the *format* rather than about the run is out here, where go vet, the linter and a table of values against their exact text can all reach it.

Types

type Answer

type Answer struct {
	// Decoded is what the generated reader made of the entry's bytes.
	Decoded *Values `json:"decoded"`

	// Written is what a reader makes of the file the generated writer produced
	// from the records Decoded holds.
	//
	// It is absent where the reading direction did not reach the end of the
	// file: a run that stopped at a failure holds no complete set of records to
	// write back, and an entry expecting a failure is an entry about reading.
	// It is absent, too, from a runner whose generator emits no writer at all —
	// emitting one is the generator's decision (docs/ir/SPEC.md, "Writing a
	// file"), and a runner is not asked to invent the direction. What this
	// repository's own runner does with that latitude is not to take it: the Go
	// generator emits a writer, so the Go runner reports this member, and
	// [CompareAnswer] holds it to it.
	Written *Values `json:"written,omitempty"`
}

Answer is what a runner reports about one entry: what the generated reader made of the entry's bytes, and what the file the generated writer laid those records back out into reads as.

Two documents rather than one because an entry states one set of values and a generator is asked about it twice, once in each direction (#68). A generator that reads a file correctly and writes one nobody can read back is a generator half the ecosystem cannot use, and a corpus that only ever read would call it conformant.

Why the writing direction is checked by reading, and not by comparing bytes

The obvious check is that the bytes the writer produced are the entry's bytes, and it is the wrong one twice over.

docs/ir/SPEC.md, "Writing a file", makes byte identity a claim about a *record* and refuses to make it about a file: under an optional terminator a writer emits a final delimiter the input need not have carried, and under segmented framing it lays a record into as few segments as the largest allows, whatever the input did. Both are deliberate, so a corpus demanding the input's bytes back would fail two of the four framings by design.

It is wrong at the field level too, and the corpus already holds the case: packed-ascii carries the lenient sign nibble A, which a reader admits as positive and a writer has no reason to emit — it writes the C the convention prescribes. The same holds of every encoding that admits more than one spelling of one value. Demanding the bytes back would make those entries unpassable by a correct generator, and dropping them would cost the corpus exactly the vectors it was seeded from.

What the specification does make normative of a file is that "a file a writer produces MUST be one that a reader built from the same descriptor reads back as the records the writer was given", so that is what is compared: the records read out of the written file, against the same values.json the reading direction is held to. It holds for all four framings and for every encoding, and it needs nothing of a runner that the reading direction did not already need.

func ParseAnswer

func ParseAnswer(b []byte) (*Answer, error)

ParseAnswer reads a runner's answer, holding it to the shape the corpus format states.

Unknown fields are refused at every level, for the reason entry.json refuses one: a key an author wrote in the expectation that it means something is a typo, and a document that silently ignores it is a document that passes for the wrong reason.

type Entry

type Entry struct {
	// Name is the entry's directory name, which is what every diagnostic about
	// it names and what a failing run reports.
	Name string

	// Dir is the directory the entry was read from.
	Dir string

	// Description is the one line entry.json carries: what shape of file this
	// entry is about.
	Description string

	// Source is the section of a specification the entry was derived from,
	// exactly as entry.json spells it. It is carried so that a failure names
	// where the expected answer came from as well as which entry disagreed
	// with it (#68).
	Source string

	// Status is whether the corpus stands behind the expected answer. An entry
	// that declares none is [Normative]; read it through
	// [Entry.IsProvisional] rather than comparing it, so that an Entry nothing
	// loaded is normative rather than exempt.
	Status Status

	// Layout is the path to the entry's layout file.
	Layout string

	// Copybooks are the paths to the copybooks in the entry's directory, sorted
	// by name. Which of them the layout names is the layout's business; what is
	// asserted here is only that an entry carries at least one.
	Copybooks []string

	// Descriptor is the IR the entry expects the layout and its copybooks to
	// resolve to, and the IR a runner's generator is handed.
	Descriptor *irpb.Descriptor

	// Input is the bytes of the file the entry describes.
	Input []byte

	// Values is what those bytes decode to.
	Values *Values
}

Entry is one corpus entry: the tuple, loaded, and the metadata saying what it is for.

The layout and the copybooks are paths rather than parsed values. Nothing here resolves a copybook — that needs the pipeline the CLI carries (#148) — and a member this package parsed but could not check would claim a coverage the loader does not have.

func Load

func Load(dir string) ([]*Entry, error)

Load reads every entry of the corpus rooted at dir, in ascending name order.

Every entry is loaded, and every entry that could not be is reported, rather than the first: a corpus with three bad entries is one editing session, not three.

A corpus holding no entry is an error. An empty directory and a corpus that failed to be found are the same thing to a caller about to report that everything passed, and only one of them is honest.

func LoadEntry

func LoadEntry(dir string) (*Entry, error)

LoadEntry reads one entry from its directory.

Everything wrong with the entry is reported at once. An entry is written by hand, so it goes wrong the way a layout does — in several places in one sitting — and a loader stopping at the first fault is a loader run once per fault.

func (*Entry) IsNormative

func (e *Entry) IsNormative() bool

IsNormative is whether the corpus stands behind the entry's expected answer, and so whether it counts and can fail a run.

It exists so that the positive question has a guarded reader too. Status is a string type whose zero value is neither of its constants, so a caller switching on the field directly would match nothing for an Entry nothing loaded — and the direction that silently matches nothing is the direction this whole member has to fall the right way in.

func (*Entry) IsProvisional

func (e *Entry) IsProvisional() bool

IsProvisional is whether the entry's expected answer is uncorroborated, and so whether a disagreement with it is reportable rather than a failure.

It is a method rather than a comparison at each of its callers so that one place decides what an unwritten status means. An Entry that never went through LoadEntry — a zero value, or one a test built — carries the empty status, and every reading of it has to agree that it is normative.

type EntryError

type EntryError struct {
	// Entry is the entry's directory name.
	Entry string

	// Err is what is wrong with it, which is [errors.Join]'s value where more
	// than one thing is.
	Err error
}

EntryError is a fault in one entry, naming the entry it is in.

The entry's name leads because a corpus failure is read against a directory listing: what an author needs first is which entry to open. Everything wrong with one entry is one of these, carrying a joined error, rather than several — an entry is a tuple and it is loaded or it is not.

func (*EntryError) Error

func (e *EntryError) Error() string

func (*EntryError) Unwrap

func (e *EntryError) Unwrap() error

type MismatchError

type MismatchError struct {
	// Entry is the entry's directory name.
	Entry string

	// Source is what the entry cites as the origin of its expected answer.
	Source string

	// Err is the disagreement, which is [errors.Join]'s value where a runner
	// disagreed in more than one place.
	Err error
}

MismatchError is a runner's answer disagreeing with what an entry expects.

It carries the entry and the source the entry cites as well as the disagreement, because a corpus failure is read by somebody who has to decide whether their generator is wrong or the entry is, and that decision starts at the section the expected answer was derived from (#68).

func (*MismatchError) Error

func (e *MismatchError) Error() string

func (*MismatchError) Unwrap

func (e *MismatchError) Unwrap() error

type PathError

type PathError struct {
	// Path is where in the values document the disagreement is, in the spelling
	// [Compare] uses throughout: `record 1 ORDER-RECORD.ORDER-ID`, with an
	// occurrence written `[0]` and counted from zero as the document writes it.
	Path string

	// Err is the disagreement, whose message already names Path.
	Err error
}

PathError is one disagreement Compare found, carrying the path through the record it is at as well as the sentence a reader sees.

The message is the sentence and nothing else: the path is already in it, spelled for whoever is reading a report, and a wrapper that said it twice would be a wrapper every report had to be edited around. What the type adds is the same path in a form a caller can key on, so that a caller holding the descriptor can say where the descriptor puts the item that disagreed — its offset, its width, its usage and its charset — beside the disagreement (#199).

That caller is the engine, and it is the only thing in the system that can do it: the adapter was never told what was expected, and this package compares documents rather than bytes. Recovering the path by parsing the sentence back out of an error string would work exactly until somebody improved the sentence, which is the kind of coupling a type costs nothing to replace.

func (*PathError) Error

func (e *PathError) Error() string

func (*PathError) Unwrap

func (e *PathError) Unwrap() error

type Record

type Record struct {
	// Name is the record's name as the copybook spells it — the `original` of
	// the record node's names, never an identifier munged from it.
	Name string `json:"name"`

	// Value is what the record's top-level node holds, in the shape
	// docs/conformance/SPEC.md's "The value language" describes: an object for
	// a group, an array for an item that repeats, and a scalar for an
	// elementary item.
	Value any `json:"value"`
}

Record is one record of a file: which record type it is, and what it holds.

type RunError

type RunError struct {
	// Entry is the entry's directory name.
	Entry string

	// Source is what the entry cites as the origin of its expected answer.
	Source string

	// Err is what stopped the run.
	Err error
}

RunError is an entry a runner could not answer about at all: the generator would not run, what it produced would not compile, or the runner could not be driven.

It carries the entry and its source for the reason MismatchError does, and it is a separate type because the two send a reader somewhere different. A mismatch is a disagreement about bytes, and whoever reads it decides whether the generator or the entry is wrong. This one is the corpus failing to ask the question, so nothing has been learned about either — and a run that reported it as a disagreement would have a generator author reading a spec section about a claim that was never tested (#68).

func (*RunError) Error

func (e *RunError) Error() string

func (*RunError) Unwrap

func (e *RunError) Unwrap() error

type Status

type Status string

Status is how much authority an entry's expected answer carries.

Every other member of an entry is a claim about a file; this one is a claim about the entry itself, and it exists because the rule that an entry is authored from a specification and never recorded from a run of the code it checks (#67) has a mirror. An entry recorded from its subject passes forever, including through the bug it was written to catch — and an entry authored from a misreading fails forever, telling every implementation it is wrong. The exposure is worst where the corpus is most valuable: an entry covering a construct no implementation handles yet has nothing to disagree with it, so review of a hand-computed byte string is the only defence and review of one is weak.

A status is not part of the question. A runner is asked about a provisional entry in exactly the words it is asked about a normative one and is never told which it was given, because an adapter that could tell would be an adapter whose answers to the uncorroborated half of the corpus are worth less than its answers to the rest.

const (
	// Normative is an entry the corpus stands behind: it counts, and a
	// disagreement with it is a failure.
	//
	// It is what an entry declaring no status is, which is the safe direction
	// for the default to fall in — a status somebody forgot to write, or one a
	// reader of this package never set, leaves the entry counting rather than
	// exempt. The other default would let an entry drop out of the verdict by
	// omission, and nothing in a passing run would say so.
	Normative Status = "normative"

	// Provisional is an entry whose expected answer nothing has corroborated
	// yet: it runs, its result is reported, and it counts in no total and
	// produces no failure verdict.
	//
	// What promotes one is written down in docs/conformance/SPEC.md, "A
	// provisional entry", and it is a fact about the world rather than
	// anything this package can check — a second implementation agreeing, or a
	// second person re-deriving the answer from the specification. Promotion is
	// therefore an edit to entry.json and a reviewer, which is the point: the
	// corpus gains somewhere to put an entry whose expected answer is contested
	// rather than the two options of shipping it as normative or not shipping
	// it at all.
	Provisional Status = "provisional"
)

type Values

type Values struct {
	// Records are the records read, in file order.
	Records []Record `json:"records"`

	// Failure is present where reading stopped at a record the file does not
	// carry correctly, and absent where the file was read to its end.
	//
	// The text is a note for whoever reads the report, and it is deliberately
	// not compared: a diagnostic is a generator's own wording in its own
	// language, so an entry demanding particular words would be an entry only
	// one generator could pass. What is compared is that a failure happened,
	// and that it happened after the records the entry lists.
	Failure string `json:"failure,omitempty"`
}

Values is what a file's bytes decode to: the records in the order they come out of the file, and whether reading stopped at a failure.

It is both halves of the comparison — what an entry's values.json states and what a runner writes on standard output — because a runner and an entry that spoke different dialects would need a translation nobody could test. See docs/conformance/SPEC.md, "What a runner does", for the whole of the contract this type is the Go reading of.

func ParseValues

func ParseValues(b []byte) (*Values, error)

ParseValues reads a values document, holding it to the shape the corpus format states.

Unknown fields are refused at both levels, for the reason entry.json refuses one: a key an author wrote in the expectation that it means something is a typo, and a document that silently ignores it is a document that passes for the wrong reason.