Documentation
¶
Overview ¶
Package schema is a registry for the estate's message schemas: it stores them under permanent identifiers that refuse to change, and resolves them offline.
What a registry is for ¶
A stored message carries a dataschema URN. Three years later somebody has to answer "what shape was this?", and the answer must be the same one a consumer would have got the day the message was written. That is the property this module exists to provide, and ErrImmutable is the whole of how it provides it — a published identifier may be republished with identical bytes, and never with different ones.
Offline first ¶
A service embeds its schemas and validates without a network call:
//go:embed schemas/orders.created.v3.json
var ordersCreated []byte
reg, err := schema.Embed(schema.Schema{
URN: urn.Schema{Name: "orders.created", Version: 3},
Language: schema.JSONSchema,
Bytes: ordersCreated,
})
A service that must reach the registry to validate has taken a runtime dependency on it, which is what a registry should most avoid being: it would then sit in the path of every message its consumers handle. The network resolver is for the case a service meets a URN it does not hold, which is the interesting case and the rare one.
Failing open, loudly ¶
A Guard that cannot resolve a URN processes the message unvalidated and increments a counter. Failing closed there would hand the registry the power over its consumers that embedding exists to deny it, turning a registry outage into an outage of everything downstream.
The counter is the load-bearing half. A log line about an unvalidated message is invisible until somebody goes looking; a metric that is normally zero and suddenly is not is the difference between a known degradation and a silent one.
A schema that resolves and does not match is a different thing entirely, and that fails closed.
It does not own a server ¶
gitlab.com/phpboyscout/go/transport already provides HTTP, gRPC and gateway servers with health endpoints and lifecycle. This module supplies handlers; schema/serve wires them. Nothing here builds a listener.
What it does not do yet ¶
It does not check backward, forward or full compatibility, and it does not claim to. The gap is shaped rather than left open: Registry.Publish takes a Compatibility from the first release and refuses anything but CompatibilityNone, a Checker exists per schema language, and the store retains prior versions so a checker will have a predecessor to compare against. Until one exists, a version bump asserts only that the bytes differ. It is not a compatibility claim.
Index ¶
- Variables
- type Checker
- type Compatibility
- type Guard
- type GuardOption
- type Language
- type MemoryStore
- type NoCheck
- type Registry
- func (r *Registry) List(ctx context.Context) ([]urn.Schema, error)
- func (r *Registry) Publish(ctx context.Context, s Schema, mode Compatibility) error
- func (r *Registry) Resolve(ctx context.Context, id urn.Schema) (Schema, error)
- func (r *Registry) Versions(ctx context.Context, name string) ([]int, error)
- type Resolver
- type Schema
- type Store
- type Validator
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrImmutable is returned when an identifier is republished with different // bytes. // // This is the property the whole registry rests on. Without it, a consumer // that validated a message yesterday cannot reproduce that result today, // and every stored dataschema becomes a claim about something that has // since moved. // // Republishing identical bytes is accepted, because a deployment that // registers its schemas on every start should be idempotent rather than // clever. ErrImmutable = errors.NewSentinel("schema.immutable", "schema: a published identifier cannot be republished with different bytes") // ErrNotFound is returned when an identifier has never been published. ErrNotFound = errors.NewSentinel("schema.not_found", "schema: no schema is published under that identifier") // ErrUnsupportedCompatibility is returned when a caller asks for a // compatibility guarantee this release cannot provide. // // The mode argument exists from the first release precisely so that this // error can exist. A caller asking for backward compatibility is told that // nothing checks it, rather than being quietly given no check at all — and // when a checker arrives, no call site changes signature to use it. ErrUnsupportedCompatibility = errors.NewSentinel("schema.unsupported_compatibility", "schema: only CompatibilityNone is supported; nothing checks compatibility yet") // ErrUnknownLanguage is returned when a schema names a language this // registry has no checker for. ErrUnknownLanguage = errors.NewSentinel("schema.unknown_language", "schema: no checker is registered for that schema language") // ErrEmpty is returned when a schema carries no bytes. An empty schema // validates everything, which is indistinguishable from validating nothing // and is never what anybody meant. ErrEmpty = errors.NewSentinel("schema.empty", "schema: a schema must carry bytes") // ErrSchemaUnusable is returned when a schema resolved but cannot be used // to validate anything — it will not compile, it is not a descriptor set, // or it does not name its root message. // // It exists so a [Guard] can tell "your message is wrong" from "our schema // is wrong", which are different failures with different owners. Without // the distinction a mis-published schema fails CLOSED and uncounted: every // message on that topic rejected while the fail-open counter reads zero and // the alert says the consumer is healthy. That is the exact outage the // fail-open design exists to prevent, arriving through the failure most // likely to actually happen. ErrSchemaUnusable = errors.NewSentinel("schema.unusable", "schema: the schema resolved but cannot be used to validate") // ErrInvalidLanguage is returned when a schema names a language that is not // one this module defines. // // A one-character typo in a Language constant would otherwise produce a // service that publishes cleanly, starts cleanly and validates nothing for // the life of the deployment, signalled only by a counter that looks // identical to a registry outage. ErrInvalidLanguage = errors.NewSentinel("schema.invalid_language", "schema: not a schema language this module defines") // ErrInvalidPayload is returned when a payload does not conform to the // schema it names. // // This is the one failure that is NOT open. Failing open is for a schema // that could not be resolved; a schema that resolved and did not match is // a real answer, and passing it on would make the whole exercise // decorative. ErrInvalidPayload = errors.NewSentinel("schema.invalid_payload", "schema: payload does not conform to its schema") )
Refusals, as sentinels so that a caller can tell "this schema does not exist" from "this schema exists and you may not change it", which are different conversations with different people.
Functions ¶
This section is empty.
Types ¶
type Checker ¶
type Checker interface {
// Language is the schema language this checker understands.
Language() Language
// Check reports whether next may replace prior under the given mode. It is
// called only when a predecessor exists.
Check(ctx context.Context, prior, next Schema, mode Compatibility) error
}
Checker decides whether a new version of a schema is compatible with its predecessor.
No real implementation exists yet — NoCheck is registered for each language instead. The interface is here from the first release so that the eventual work is an implementation rather than a refactor of every call site, and so the seam has been travelled before something real travels it.
type Compatibility ¶
type Compatibility string
Compatibility is the guarantee a publisher asks for.
Only CompatibilityNone is accepted today, and the others exist so that a caller who wants a real guarantee is refused rather than quietly given nothing. When a checker arrives, the values start meaning what they say and no call site changes.
const ( // CompatibilityNone asks for no check. It is the only accepted value. CompatibilityNone Compatibility = "none" // CompatibilityBackward would mean a consumer of the previous version can // read data written against this one. CompatibilityBackward Compatibility = "backward" // CompatibilityForward would mean a consumer of this version can read data // written against the previous one. CompatibilityForward Compatibility = "forward" // CompatibilityFull would mean both. CompatibilityFull Compatibility = "full" )
The compatibility modes. Only None is accepted; see ErrUnsupportedCompatibility.
type Guard ¶
type Guard struct {
// contains filtered or unexported fields
}
Guard validates a payload against the schema its dataschema names, and fails open when it cannot resolve one.
The asymmetry is the whole design:
- Could not resolve the schema — unknown URN, not a URN at all, no validator for its language, registry unreachable. **Fails open**, and increments a counter.
- Resolved the schema and the payload does not match. **Fails closed**, with ErrInvalidPayload.
Failing open on the first is D7 of the spec: the embedded register means the common case never touches the network, so the failing case is already the rare one. Failing closed there would hand the registry the power over its consumers that embedding exists to deny it, and turn a registry outage into an outage of everything downstream of it.
Example (FailOpen) ¶
ExampleGuard_failOpen shows the asymmetry that makes a registry outage survivable: a schema that cannot be resolved does not stop the message, and the counter is how anybody finds out.
package main
import (
"context"
"fmt"
"gitlab.com/phpboyscout/go/cloudevents/urn"
"gitlab.com/phpboyscout/go/schema"
"gitlab.com/phpboyscout/go/schema/jsonschema"
)
const orderSchema = `{
"type": "object",
"required": ["order"],
"properties": {"order": {"type": "string"}}
}`
func main() {
reg, err := schema.Embed(schema.Schema{
URN: urn.Schema{Name: "orders.created", Version: 3},
Language: schema.JSONSchema,
Bytes: []byte(orderSchema),
})
if err != nil {
fmt.Println("embed:", err)
return
}
guard := schema.NewGuard(reg, []schema.Validator{jsonschema.New()})
// Version 9 is not embedded and there is no network resolver behind this
// guard, so it cannot be validated.
err = guard.Validate(context.Background(), "urn:phpboyscout:schema:orders.created:9", []byte(`{}`))
fmt.Println("the message was processed:", err == nil)
fmt.Println("and it was counted:", guard.Unvalidated())
}
Output: the message was processed: true and it was counted: 1
func NewGuard ¶
func NewGuard(resolver Resolver, validators []Validator, opts ...GuardOption) *Guard
NewGuard builds a guard over a resolver and a validator per language.
func (*Guard) Unvalidated ¶
Unvalidated is how many payloads this guard has processed without validating them.
It is a counter and not a log line on purpose. A log line about an unvalidated message is invisible until somebody goes looking; a number that is normally zero and is suddenly not is the difference between a known degradation and a silent one. Alert on it.
type GuardOption ¶
type GuardOption func(*Guard)
GuardOption configures a Guard.
func OnFailOpen ¶
func OnFailOpen(f func(ctx context.Context, dataschema string, cause error)) GuardOption
OnFailOpen registers a callback invoked every time the guard processes a payload it could not validate.
This is where an OpenTelemetry counter is wired. It is a callback rather than a metric instrument so that this module does not put an observability stack into the graph of every service that reads a message — the counter is the requirement, the SDK is the caller's choice.
type Language ¶
type Language string
Language names the form a schema is written in.
Both are first-class. JSON Schema is the natural fit for CloudEvents' JSON payloads; protobuf descriptors matter because gRPC services in this estate would register too, and a registry serving only one would send the other somewhere else. The cost, stated plainly, is that everything language-specific doubles — checking most of all.
The schema languages this registry addresses.
func (Language) Valid ¶
Valid reports whether a language is one this module defines.
Exported because it is the check a caller building a Schema from a config file or an HTTP header needs, and because the alternative — comparing against the constants by hand at every call site — is where the typo gets in.
type MemoryStore ¶
type MemoryStore struct {
// contains filtered or unexported fields
}
MemoryStore holds schemas in memory. It is the embedded register a service registers its own schemas into at start-up, and it is also a perfectly good backing store for a single-instance registry.
It keeps every version it is given, forever. That is Store's contract rather than this implementation's convenience: a compatibility checker needs a predecessor to compare against, and a store that kept only the latest would make that retrofit require a backfill nobody has the data for.
func (*MemoryStore) List ¶
List returns every published identifier, ordered by name and then version so that two calls against an unchanged store agree.
func (*MemoryStore) Put ¶
func (m *MemoryStore) Put(_ context.Context, s Schema) error
Put publishes a schema.
Republishing identical bytes under the same identifier succeeds, so that a service registering its schemas on every start is idempotent. Republishing different bytes is ErrImmutable, and that refusal is the property the whole registry rests on.
func (*MemoryStore) Versions ¶
Versions returns every published version of a name, ascending. A name nothing has been published under returns an empty slice rather than an error: "no versions" is an answer, and callers that treat it as a failure end up special-casing the first publication of every schema.
type NoCheck ¶
type NoCheck struct {
Lang Language
}
NoCheck is a Checker that accepts everything, registered for each language until a real one exists.
It is not a placeholder to be deleted. It is what makes the mode argument and the checker lookup live code today, so the path a real checker will take has been exercised before a real checker takes it.
func (NoCheck) Check ¶
Check accepts everything. Publishing still refuses any mode but CompatibilityNone before reaching here, so this is never asked a question it would have to lie about.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry is the registry proper: a Store plus a Checker per language.
Publishing and reading are deliberately different methods on it rather than one surface, because the asymmetry shapes everything above: read paths are safe to expose broadly, write paths are not, and they should not look alike to whoever is wiring authentication in front of them.
func Embed ¶
Embed builds a registry over an in-memory store holding the given schemas, with a no-op checker for each language present.
This is the client half, and it is not an afterthought. A service that must reach the registry to validate has taken a runtime dependency on it, which is the thing a registry should most avoid being — it would then sit in the path of every message its consumers handle.
//go:embed schemas/orders.created.v3.json
var ordersCreated []byte
reg, err := schema.Embed(schema.Schema{
URN: urn.Schema{Name: "orders.created", Version: 3},
Language: schema.JSONSchema,
Bytes: ordersCreated,
})
Example ¶
ExampleEmbed is the shape a service uses: register what you hold at start-up, and validate without a network call.
package main
import (
"context"
"fmt"
"gitlab.com/phpboyscout/go/cloudevents/urn"
"gitlab.com/phpboyscout/go/errors"
"gitlab.com/phpboyscout/go/schema"
"gitlab.com/phpboyscout/go/schema/jsonschema"
)
const orderSchema = `{
"type": "object",
"required": ["order"],
"properties": {"order": {"type": "string"}}
}`
func main() {
reg, err := schema.Embed(schema.Schema{
URN: urn.Schema{Name: "orders.created", Version: 3},
Language: schema.JSONSchema,
Bytes: []byte(orderSchema),
})
if err != nil {
fmt.Println("embed:", err)
return
}
guard := schema.NewGuard(reg, []schema.Validator{jsonschema.New()})
fmt.Println("conforming:", guard.Validate(context.Background(),
"urn:phpboyscout:schema:orders.created:3", []byte(`{"order":"4172"}`)))
fmt.Println("missing a required field:", errors.Is(guard.Validate(context.Background(),
"urn:phpboyscout:schema:orders.created:3", []byte(`{}`)), schema.ErrInvalidPayload))
fmt.Println("unvalidated so far:", guard.Unvalidated())
}
Output: conforming: <nil> missing a required field: true unvalidated so far: 0
func New ¶
New builds a registry over a store.
A checker is registered per language. Pass NoCheck for each until real ones exist — New refuses a schema language it has no checker for at publish time, so an unregistered language is a wiring mistake found at the first publication rather than a schema that silently skips a check somebody thinks is running.
func (*Registry) Publish ¶
Publish stores a schema under its identifier.
The mode argument is required from the first release even though nothing checks it. CompatibilityNone is the only accepted value; anything else is ErrUnsupportedCompatibility. A caller asking for a guarantee we cannot provide is told so, rather than quietly given none — and when a checker arrives, no call site changes signature to use it.
Example ¶
ExampleRegistry_Publish shows the immutability refusal, which is the property every stored dataschema depends on.
package main
import (
"context"
"fmt"
"gitlab.com/phpboyscout/go/cloudevents/urn"
"gitlab.com/phpboyscout/go/errors"
"gitlab.com/phpboyscout/go/schema"
)
const orderSchema = `{
"type": "object",
"required": ["order"],
"properties": {"order": {"type": "string"}}
}`
func main() {
ctx := context.Background()
reg := schema.New(schema.NewMemoryStore(), schema.NoCheck{Lang: schema.JSONSchema})
s := schema.Schema{
URN: urn.Schema{Name: "orders.created", Version: 3},
Language: schema.JSONSchema,
Bytes: []byte(orderSchema),
}
fmt.Println("first publication:", reg.Publish(ctx, s, schema.CompatibilityNone))
fmt.Println("republished identically:", reg.Publish(ctx, s, schema.CompatibilityNone))
changed := s
changed.Bytes = []byte(`{"type":"string"}`)
fmt.Println("republished differently:",
errors.Is(reg.Publish(ctx, changed, schema.CompatibilityNone), schema.ErrImmutable))
fmt.Println("asking for a guarantee nothing checks:",
errors.Is(reg.Publish(ctx, s, schema.CompatibilityBackward), schema.ErrUnsupportedCompatibility))
}
Output: first publication: <nil> republished identically: <nil> republished differently: true asking for a guarantee nothing checks: true
func (*Registry) Resolve ¶
Resolve returns the schema published under an identifier, or ErrNotFound.
type Resolver ¶
Resolver turns an identifier into a schema. Registry is one; so is the HTTP client. A Guard takes this rather than a Store because it only ever reads, and because reading is the half that is safe to expose broadly.
type Schema ¶
type Schema struct {
URN urn.Schema
Language Language
Bytes []byte
// Root names the entry point within Bytes, for languages that need one.
//
// A JSON Schema document is its own root and ignores this. A protobuf
// FileDescriptorSet is not: it holds every message the target file depends
// on, and nothing in it says which one a payload of this schema is. Root
// carries the fully-qualified message name — "uk.phpboyscout.OrderCreated".
//
// It is a language-specific field on a language-agnostic struct, which is
// not lovely. The alternative was inferring the entry point by convention
// (the first message of the last file, say), and a convention that is wrong
// once produces a validator silently checking the wrong message.
Root string
}
Schema is one version of one schema: an identifier, a language, and the bytes. The bytes are never interpreted here — that is a Validator's job, and it lives in a sub-package so a JSON-only consumer does not take a protobuf runtime.
type Store ¶
type Store interface {
// Get returns the schema published under an identifier, or [ErrNotFound].
//
// The returned Schema.Bytes MUST NOT alias the store's own copy. A caller
// that can mutate what it reads can rewrite a published schema, which
// defeats [ErrImmutable] from the read side and races every other reader.
Get(ctx context.Context, id urn.Schema) (Schema, error)
// Put publishes a schema. It MUST return [ErrImmutable] if the identifier
// already carries different bytes, and MUST succeed if it carries
// identical ones.
Put(ctx context.Context, s Schema) error
// Versions returns every published version of a name, ascending. It returns
// an empty slice, not an error, for a name nothing has been published
// under.
//
// Ascending is part of the contract rather than an accident of insertion
// order: a compatibility checker looks for the highest version below the
// one being published, and the HTTP listing is served straight from this.
Versions(ctx context.Context, name string) ([]int, error)
// List returns every published identifier.
List(ctx context.Context) ([]urn.Schema, error)
}
Store is where schemas live. One seam, and it has to exist anyway so that an embedded register and a network-backed one are the same thing to a caller.
An implementation MUST retain prior versions under the same name. That is not an optimisation: a compatibility checker needs a predecessor to compare against, and a store that kept only the latest would make that retrofit require a backfill nobody has the data for.
type Validator ¶
type Validator interface {
// Language is the schema language this validator understands.
Language() Language
// Validate returns [ErrInvalidPayload] when the payload does not conform.
Validate(ctx context.Context, s Schema, payload []byte) error
}
Validator checks a payload against a schema. Implementations live in sub-packages, one per language, so that the dependency each needs lands only in the graph of a service that uses that language.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package client resolves schemas over HTTP, for the case a service meets an identifier it does not hold.
|
Package client resolves schemas over HTTP, for the case a service meets an identifier it does not hold. |
|
Package httpapi is the registry's HTTP surface: two handlers, one for reading and one for publishing.
|
Package httpapi is the registry's HTTP surface: two handlers, one for reading and one for publishing. |
|
Package jsonschema validates JSON payloads against JSON Schema documents.
|
Package jsonschema validates JSON payloads against JSON Schema documents. |
|
Package protobuf validates protobuf payloads against a FileDescriptorSet.
|
Package protobuf validates protobuf payloads against a FileDescriptorSet. |
|
Package serve wires the registry's handlers onto a go/transport HTTP server.
|
Package serve wires the registry's handlers onto a go/transport HTTP server. |