Documentation
¶
Overview ¶
Package conceptfields reads what every concept in the DSL tree DECLARES -- its top-level field names, which of them are required, and the values of every enum -- and holds the committed snapshot of that answer.
=========================================================================== WHY A SNAPSHOT EXISTS AT ALL (memql#5209) =========================================================================== Removing a field from a concept BRICKS every stored row that carries it. Every concept builds its JSON schema with `additionalProperties: false` and a mutation's read-merge validates the MERGED payload, stored keys included, so the next write to such a row is refused with `additionalProperties '<field>' not allowed`. memql#5199 wrote the gate for the DESTRUCTIVE direction -- a migration that strips too widely -- and could not cover this one, because a retirement that shipped no migration leaves nothing for a migration-reader to read.
The check that WOULD cover it is "did a field disappear from a concept in this branch", and that needs a BEFORE. `git merge-base` cannot supply one: every `actions/checkout` in ci.yml is depth-1, so the Go lanes have no history at all. A committed snapshot IS the merge base by construction -- the file in the tree is what main has -- and it works in a depth-1 checkout, in `make test`, with no database and no network.
=========================================================================== THE SOURCE IS THE BUILT SCHEMA, NOT A SECOND PARSE OF THE TEXT =========================================================================== Scan runs the loader's own extractor (memql.ExtractConceptDecls, which slices concept blocks out and hands each to component/language/parser) and the real schema builder, then reads the fields out of the emitted JSON Schema. That is deliberate, and it is the whole reason to prefer it over the regex reader memql#5199 shipped:
- The snapshot's claim is exactly "these are the keys a row may carry", and the artifact that decides that at runtime is this same schema. A regex over the source answers a similar question, not the same one.
- `@required` is not one spelling. It is the `!` sigil AND the annotation, folded by propertyDeclToParsed. A second implementation of that fold is a second answer to "is this field required", live in a gate whose whole job is to notice when the answer changes.
- Enum values live on the TypeRef, and array-of-enum, defaults and variants each shape the emitted schema in ways a line-oriented reader does not see.
=========================================================================== TOP-LEVEL FIELDS ONLY, AND SAYING SO IS PART OF THE GATE =========================================================================== A nested object's members are not top-level payload keys, and `payload - 'x'` -- the only strip form a migration writes -- cannot reach one. So the snapshot records top-level fields and the ledger governs top-level retirements. A narrowing INSIDE a nested block is a real hazard of the same family and this file does not cover it; that is written down here rather than left for a reader to discover, because a gate that hides what it cannot examine is worse than no gate -- an auditor seeing one stops looking.
Index ¶
- Constants
- func DefaultRoots() []string
- func Marshal(s Snapshot) ([]byte, error)
- func Reconcile(committed, current Snapshot) (Snapshot, []Narrowing)
- func SQLStatements(sql string) []string
- func ScopedConcepts(stmt string) []string
- func StrippedPayloadKeys(stmt string) []string
- func VerifyLedger(snap Snapshot, migrationsDir string) []string
- type Entry
- type File
- type Narrowing
- type Result
- type Retirement
- type Snapshot
- type Strip
Constants ¶
const ( // KindField -- a field disappeared from a concept. The stored key is now // `additionalProperties '<field>' not allowed`. KindField = "field" // KindEnumValue -- an enum lost a value. A row carrying it stops // validating, and the message names the enum. KindEnumValue = "enumValue" // KindRequired -- a field became required, or arrived required on a // concept that already had rows. Existing rows lack it, so the read-merge // fails the other way round. KindRequired = "required" // KindConcept -- the concept itself is gone from the tree. // // NOT a bricking hazard: nothing writes a concept the engine no longer // registers, so its rows are inert rather than broken. It is in the ledger // because of the OTHER way a concept leaves the snapshot -- an id that // DRIFTED. A `namespace.pin` added to a directory, or a change in how ids // are assembled, silently re-keys every concept under it, and a diff that // only compared concepts present on both sides would read that as nothing // at all while the snapshot quietly stopped describing the tree. Requiring // a line per departure is what makes such a drift stop the build. KindConcept = "concept" )
The three shapes a concept can narrow in, plus the one that takes a whole concept away. Every one of them makes some stored row fail validation on its next write, at the same line, with a message that points at the schema rather than at the row's history -- which is why they are one ledger and not four.
const ( DefaultDSLRoot = "dsl" DefaultMigrationsDir = "component/database/memory-nodes/migrations" // DefaultPackGlob finds every STOREFRONT PACK's DSL tree (epic // memql#5532, issue memql#5549). // // PACKS ARE SNAPSHOTTED BECAUSE THEY SHIP NOW. While a pack lived under // examples/ behind a build tag no published image set, its concepts // could hold no rows on any cluster and there was nothing for this gate // to protect. Linking the storefront packs into the default build makes // their rows as real as any core concept's, so dropping a field from // v1:reviews:review would brick stored rows exactly as dropping one // from a core concept would -- silently, because CI's db-tests run on a // fresh database. // // A GLOB RATHER THAN A LIST, so the wholesale pack and whatever follows // it are covered the day they land rather than the day somebody // remembers this file. DefaultPackGlob = "packs/*/dsl" )
DefaultDSLRoot and DefaultMigrationsDir are the two trees the snapshot is derived from and checked against.
const DefaultSnapshotPath = "component/conceptfields/concept-fields.snapshot.json"
DefaultSnapshotPath is where the committed snapshot lives, relative to the repository root.
Variables ¶
This section is empty.
Functions ¶
func DefaultRoots ¶ added in v0.22.9
func DefaultRoots() []string
DefaultRoots is the core tree plus every pack tree on disk, in a stable order. A missing packs/ directory simply contributes nothing.
func Marshal ¶
Marshal renders a snapshot to the exact bytes the file should hold.
One rendering, used by both the writer and the drift check, so `--check` can never disagree with what a write would have produced -- the failure mode where a gate reports drift that regenerating does not fix.
func Reconcile ¶
Reconcile builds the snapshot that should be committed, and reports every narrowing the ledger does not yet cover.
The committed ledger is carried forward VERBATIM and the current tree supplies the concepts, so a regeneration is idempotent: running it twice produces the same bytes. Unrecorded narrowings are returned rather than written, which is what makes the generator REFUSE rather than quietly absorb the removal it exists to notice.
func SQLStatements ¶
SQLStatements splits on `;` after removing comments. The statement is the right unit: the concept predicate that makes a strip safe has to be in the same one, and a WHERE clause two statements away protects nothing.
func ScopedConcepts ¶
ScopedConcepts reads every concept id the statement pins itself to, handling `concept = 'x'` and `concept IN ('x', 'y')`.
EVERY id in an IN-list, not just the first. One statement can be right about one concept and wrong about the next, and a reader that stopped at the first would clear exactly that migration.
func StrippedPayloadKeys ¶
StrippedPayloadKeys reads every key a `payload - 'x'` expression removes, including the chained `payload - 'a' - 'b'` form.
func VerifyLedger ¶
VerifyLedger checks the snapshot's ledger against the migrations corpus, in BOTH directions.
Forward: an entry naming a migration must name one that exists, and -- for a field retirement -- one that actually strips that key from that concept. An entry that named a migration doing something else would read as repaired and repair nothing.
Inverse: every strip in the corpus must have an entry. This is what gives the ledger's append-only discipline something behind it: deleting an entry whose migration is still committed fails the build, so the only way to erase a retirement from the record is to delete its migration too, which is a diff nobody merges by accident.
Types ¶
type Entry ¶
type Entry struct {
Concept string `json:"concept"`
// File is the source path, carried for the diagnostic rather than for the
// comparison: a concept that MOVES between files keeps its id, and the
// snapshot must not read that as a retirement plus an addition. memql#5165
// missed `active` exactly because a per-file comparison could not see a
// field move between concepts in one file; keying by concept id is the fix,
// and File is here so the failure can name where to look.
File string `json:"file"`
Fields []string `json:"fields"`
Required []string `json:"required,omitempty"`
Enums map[string][]string `json:"enums,omitempty"`
}
Entry is one concept's declared surface, as the snapshot records it.
Every slice is sorted and every map is emitted sorted, so the committed file is a function of the tree alone -- a generator whose output depended on map iteration order would produce a fresh diff on every run and the drift gate would be unusable.
type File ¶
type File struct {
Readme []string `json:"readme"`
Concepts []Entry `json:"concepts"`
Retired []Retirement `json:"retired"`
}
File is the on-disk shape.
type Narrowing ¶
type Narrowing struct {
Concept string
// Field is empty for KindConcept.
Field string
Kind string
// Value is set for KindEnumValue only.
Value string
}
Narrowing is one thing the tree takes away that the snapshot still records.
func Narrowings ¶
Narrowings reports everything `after` takes away that `before` recorded.
A concept absent from `before` contributes nothing: it is NEW, and a new concept has no stored rows to brick, so its required fields and its enums are free. That asymmetry is the reason this is a diff against a snapshot rather than a rule about the tree.
type Result ¶
type Result struct {
// Snapshot is what should be committed: the current tree's concepts plus
// the committed ledger, carried forward.
Snapshot Snapshot
// Committed and Wanted are the bytes on disk and the bytes that should be
// on disk. ONE rendering produces both, so `--check` can never report drift
// that regenerating does not fix.
Committed []byte
Wanted []byte
// Unrecorded is every narrowing the ledger does not cover.
Unrecorded []Narrowing
// Ledger is every problem with the ledger itself, in either direction.
Ledger []string
}
Result is one evaluation of the tree against the committed snapshot.
func Verify ¶
Verify reads the tree and the committed snapshot and answers everything both the generator and the drift test need.
ONE FUNCTION, TWO CALLERS, and that is the point: cmd/conceptsnapshot and TestConceptFieldSnapshotIsNotStale ask the same question, and a gate whose answer differed from the fix command's would be a gate nobody could satisfy.
func (Result) Problem ¶
Problem renders everything wrong, or "" when nothing is.
The message is the whole user interface of this gate. Somebody meets it mid-way through a branch that felt finished, so it says what changed, why it matters, and the exact line to add -- rather than naming a rule and leaving them to find the format.
type Retirement ¶
type Retirement struct {
Concept string `json:"concept"`
Field string `json:"field,omitempty"`
Kind string `json:"kind"`
Value string `json:"value,omitempty"`
// Migration is the base name of the up-migration that repairs the stored
// rows, e.g. `20260908010000_cluster_status_retired.up.sql`.
Migration string `json:"migration,omitempty"`
// Waiver is the reason no migration is needed. Free text, and it is read
// by people rather than by code -- the gate checks that ONE was written,
// never that it is true.
Waiver string `json:"waiver,omitempty"`
// Note carries the provenance: which issue or epic took the field away.
Note string `json:"note,omitempty"`
}
Retirement is one recorded narrowing: what was taken away, and what was done about it.
EXACTLY ONE OF Migration AND Waiver IS SET, and the pair is the whole point. A retirement that genuinely needs no migration -- nothing ever wrote the field, or the concept itself is gone -- must still be RECORDED, because without a waiver the gate's only remedy is a no-op migration, and a no-op in the migrations directory is a lie told to whoever reads it next.
APPEND-ONLY. Nothing here enforces that, and saying so is deliberate: the check would need history, and every Go lane in CI checks out at depth 1 -- which is the same constraint that made this file a snapshot rather than a `git merge-base` diff. What DOES stand behind it is VerifyLedger's inverse direction: a migration that strips a key must have a ledger entry, so deleting an entry whose migration still exists fails the build. Deleting both is a diff, and a diff is what review is for.
type Snapshot ¶
type Snapshot struct {
// Concepts is sorted by id.
Concepts []Entry `json:"concepts"`
// Retired is the ledger. It is APPEND-ONLY by discipline and by review --
// see the Retirement type, which says what is and is not enforced about
// that.
Retired []Retirement `json:"retired"`
}
Snapshot is the committed answer: every concept's surface, plus the append-only ledger of everything that has been taken away from one.
func Load ¶
Load reads the committed snapshot. A missing file is an EMPTY snapshot and no error, so the very first generation has a `before` to diff against -- which is nothing, and therefore reports no narrowing. Every field in the tree on the day this lands is baseline, not a retirement.
func ScanRoots ¶ added in v0.22.9
Scan reads every concept declared under dslRoot.
IT WALKS THE TREE THE LOADER'S OWN WAY, and both halves of that matter.
- dslfs.WalkMemqlFiles is the walk, so the soft-disable convention (`_reference/`, `_`-prefixed files) is honoured by the same code that honours it at boot rather than by a second reading of the same rule.
- memql.ExtractConceptDecls is the parse. A `.memql` file is NOT a parseable whole for every construct kind -- handing an automations file to parser.ParseFile fails at the first `automation` keyword -- so the loader slices each concept block out and parses the slice. A snapshot built on the naive parse would silently hold only the files that happen to contain nothing else, which is most of `concepts.memql` and none of the interesting corners.
The whole tree rather than a `concepts.memql` glob, which is what memql#5199's reader did: `dsl/shopify/generated/` holds 65 concepts one per file, and a glob that misses them leaves the largest single population in the tree outside every check built on it. A mirror's rows brick exactly the way a native concept's do. ScanRoots scans several DSL trees as one snapshot.
The core tree plus every storefront pack's, because a pack in the default build holds rows a field drop would brick just as a core concept's does. Duplicate-id detection spans the whole set, which is what catches a pack shadowing a core concept rather than letting the walk order decide.
type Strip ¶
Strip is one (concept, key) pair an up-migration removes.
func StripsIn ¶
StripsIn reads every (concept, key) an up-migration under dir strips.
UP MIGRATIONS ONLY. A down migration legitimately strips a key the CURRENT tree declares -- 20260603000000_attachment_rename_gcsurl_to_bloburl.down.sql strips `blobUrl`, which is the live name -- so reading them would report correct code as a violation.
Source Files