store

module
v1.5.1-0...-433bfd8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 12, 2026 License: Apache-2.0

README

store

One contract, many targets. protoc-gen-store is a co-generation factory: feed it Protobuf (annotated with Google AIP) and it emits production-grade Prisma, GORM, and PostgreSQL schemas; feed it a GraphQL endpoint or .graphql SDL and it emits a typed Go client. One binary, one store.yaml, outputs that always agree.

Release Go Reference Go Version Buf BSR CI License

Contents

Overview

store is a single binary, protoc-gen-store, built as a co-generation factory — a source → target × language pipeline. A source turns a contract into a neutral model; a target renders that model into code. Two sources ship today:

  • proto — a protoc plugin. Annotate messages with the Google AIP standards you already use (google.api.resource, field_behavior, resource_reference) and store infers tables, columns, primary keys, foreign keys, and relations.
  • graphql — point it at a live GraphQL endpoint (or a cached .graphql SDL file) and it introspects the schema.

Both run through the same binary on buf generate, selected per output by a target= plugin entry; everything else lives in one store.yaml.

Source Target Output
proto prisma A complete, runnable Prisma 7 project — multi-file schema, package.json, tsconfig.json, config, and a .env.example to copy to .env.
proto gorm Go structs with GORM tags + a migration registry; optional CRUD stores + first-party telemetry, AIP-160 filter / AIP-132 order_by list engines for SQL and Hasura GraphQL, and proto ↔ model converters.
proto sql PostgreSQL DDL — per-schema reference files and one transactional, idempotent migrate.sql; FK constraints, indexes, updated_at triggers, COMMENT ON.
graphql graphql A typed Go GraphQL client — row models, a fluent predicate DSL, CRUD/subscription methods (see GraphQL client SDK).

Each database target also emits a README.md with a Mermaid ER diagram and a per-model column reference, so the generated tree is self-documenting. Postgres and MongoDB providers are both supported, and the GraphQL client speaks the Hasura/DDN/Grafbase lineage through a pluggable dialect. Code emitters are organized per language (Go today), so a target can grow Python/TypeScript/… output by adding a template set beside the shared, language-agnostic parsing.

Features

  • A co-generation factory. One source → target × language pipeline. Sources (proto, GraphQL) turn a contract into a neutral model; targets render it. Adding a target or a source is a self-contained addition — the shared parsing, config, and emit machinery are reused. The proto IR engine (protokit) is itself generic, so a sibling plugin (web3) renders a blockchain backend from the same model.
  • AIP-native. ~80% of the proto schema is read straight from standard AIP annotations; only the remaining ~20% needs entity.v1.* / store.v1.* options.
  • GraphQL client from a schema. Point the graphql target at an endpoint or a .graphql SDL file and get a typed Go client — models, a predicate DSL, CRUD and subscriptions — behind a pluggable dialect (Hasura/DDN/Grafbase today).
  • Ready for more languages. Each code target keeps its language-specific templates under a per-language folder (Go now), so Python/Rust/TypeScript output is a new template set over the same parsing — not a rewrite.
  • Production defaults. ULID surrogate keys, auto-managed timestamps, FK indexing, soft-delete markers, and enum hygiene — all overridable.
  • List queries built in — SQL and Hasura. Opt-in AIP-160 filter / AIP-132 order_by engines: generated per-resource specs drive one shared filterx SDK whose GORM (SQL) and Hasura DDN (GraphQL) engines accept identical filter strings by construction — free-text search, sort allowlists, and opaque page tokens included.
  • Idempotent SQL. The consolidated migrate.sql is transactional and guarded (IF NOT EXISTS, CREATE OR REPLACE, deferred FK ALTERs) — safe to re-apply.
  • Relational nesting. Nested and imported value messages become real child tables (PK + FK), never opaque JSONB blobs — your structure stays queryable.
  • Monorepo layout. An optional store.yaml maps proto packages to databases and schemas without per-file annotations.
  • Deterministic. Re-running on unchanged protos produces byte-identical output (enforced by golden tests), so regenerate → migrate diff is a no-op.
  • Self-documenting. Each target ships a README.md with a Mermaid ER diagram.

Architecture

The factory is a source → model → target pipeline. A source turns an input contract into a neutral model; a target renders that model. All of it lives in one binary, configured by one store.yaml.

flowchart LR
    subgraph Sources
        direction TB
        P[".proto + AIP annotations<br/>+ entity.v1 / store.v1 options"]
        Q["GraphQL endpoint<br/>or .graphql SDL"]
    end

    P --> PS["proto source<br/>(protokit IR)"]
    Q --> QS["graphql source<br/>(introspect / SDL → IR)"]
    PS --> M[("core model<br/>+ typed facets")]
    QS --> M

    M --> T{{"target × language"}}
    T --> PR["prisma"]
    T --> GO["gorm"]
    T --> SQL["sql"]
    T --> GQL["graphql<br/>(Go client)"]

    PR --> DB[("PostgreSQL /<br/>MongoDB")]
    GO --> DB
    SQL --> DB
    GQL --> APP["your Go app"]

Each source builds the model, then the selected target renders it independently. The parsing is language-agnostic — a target's language-specific templates live under a per-language folder, so adding Python/TypeScript output reuses the same model. On the proto side, files that declare the same datasource name merge into one database, so a multi-file proto package becomes a single schema tree; the proto IR is built by the generic protokit engine, which a sibling plugin (web3) reuses for a blockchain backend.

How it works

Every annotation maps to a concrete piece of schema. store collects them all into the IR, applies its production defaults, then hands the IR to the selected renderer.

flowchart TD
    R["google.api.resource<br/>on a message"] -->|"a table (schema + name)"| T["Table in IR"]
    ID["field_behavior = IDENTIFIER"] -->|"PRIMARY KEY → demoted to UNIQUE<br/>when a surrogate id is synthesized"| T
    REQ["field_behavior = REQUIRED"] -->|"NOT NULL<br/>(nullable otherwise → pointer/? types)"| T
    REF["resource_reference<br/>on a field"] -->|"FOREIGN KEY → resolved to referenced PK"| T
    SC["proto scalar / well-known type"] -->|"SQL column type"| T
    O["entity.v1 / store.v1<br/>options"] -->|"overrides + extras<br/>(types, indexes, id strategy, FK actions)"| T

    T --> IR[("IR")]
    IR --> RENDER{{"target renderer"}}
    RENDER --> PR["prisma"]
    RENDER --> GO["gorm"]
    RENDER --> SQL["sql"]
Source annotation Inferred output
google.api.resource on a message a table; schema + name from type / plural
field_behavior = IDENTIFIER PRIMARY KEY NOT NULL
field_behavior = REQUIRED NOT NULL (nullable otherwise → pointer/? types)
resource_reference on a field a FOREIGN KEY, resolved to the referenced PK
proto scalar / well-known type the column's SQL type (see Type mapping)

Install

# Homebrew (macOS / Linux)
brew install the-protobuf-project/tap/protoc-gen-store

# or go install
go install github.com/the-protobuf-project/store/plugin/cmd/protoc-gen-store@latest

Releases ship prebuilt binaries for linux / darwin / windows on amd64 / arm64 on the Releases page. The plugin must be on your PATH so protoc/buf can find it.

You'll also need the option definitions on your import path. With buf, add the module to your buf.yaml deps:

deps:
  - buf.build/the-protobuf-project/store

then import "entity/v1/annotations.proto"; and import "store/v1/annotations.proto"; in your protos.

Quick start

1. Annotate a proto.

syntax = "proto3";
package bookstore.v1;

import "google/api/field_behavior.proto";
import "google/api/resource.proto";
import "entity/v1/annotations.proto";
import "store/v1/annotations.proto";

option (entity.v1.datasource) = {
  database: "bookstore_db"
  provider: "postgres"
};

message Author {
  option (google.api.resource) = {
    type: "bookstore.v1/Author"
    pattern: "authors/{author}"
    singular: "author"
    plural: "authors"
  };
  // Use a generated ULID primary key + created_at/updated_at columns.
  option (entity.v1.table) = { id: ID_STRATEGY_ULID, timestamps: true };

  // IDENTIFIER → the AIP resource name; becomes a UNIQUE lookup column.
  string name = 1 [(google.api.field_behavior) = IDENTIFIER];

  // REQUIRED → NOT NULL; string defaults to VARCHAR(255).
  string display_name = 2 [(google.api.field_behavior) = REQUIRED];

  // Override the default for a free-form column.
  string bio = 3 [(store.v1.column) = { type: "TEXT" }];
}

2. Add the plugin to buf.gen.yaml.

version: v2
plugins:
  - local: protoc-gen-store
    out: generated/prisma
    opt: [target=prisma]   # prisma | gorm | sql

3. Generate.

buf generate

[!NOTE] store doesn't generate Go message stubs, so protoc/buf needs a Go import path for each file. If your protos don't set option go_package, supply it per file in opt: with an M mapping, e.g. Mbookstore/v1/bookstore.proto=example.com/gen/bookstore/v1.

What comes out

The Author message above produces, across targets — Prisma:

model Author {
  id          String   @id @default(ulid()) @map("id")
  name        String   @unique @map("name")
  displayName String   @map("display_name")
  bio         String?  @map("bio")
  createdAt   DateTime @default(now()) @map("created_at")
  updatedAt   DateTime @updatedAt @map("updated_at")
  books       Book[]

  @@map("authors")
  @@schema("bookstore_v1")
}

GORM:

type Author struct {
  ID          string    `gorm:"column:id;primaryKey;not null"`
  Name        string    `gorm:"column:name;not null;uniqueIndex"`
  DisplayName string    `gorm:"column:display_name;not null"`
  Bio         *string   `gorm:"column:bio"`
  CreatedAt   time.Time `gorm:"column:created_at;autoCreateTime"`
  UpdatedAt   time.Time `gorm:"column:updated_at;autoUpdateTime"`
  Books       []Book    `gorm:"foreignKey:AuthorID"`
}

func (*Author) TableName() string { return "bookstore_v1.authors" }

(/// doc comments and json/validate tags are emitted too — trimmed here for space.)

And every target also drops a README.md with the relationships drawn out, e.g.:

erDiagram
    AUTHOR ||--o{ BOOK : writes
    AUTHOR {
        string id PK
        string name UK
        string display_name
        string bio
        timestamptz created_at
        timestamptz updated_at
    }
    BOOK {
        string id PK
        string author_id FK
        string title
    }

Output layout

Files that declare the same datasource name merge into one database, so a multi-file proto package becomes a single schema tree. Each target lays its output out to match:

generated/prisma/bookstore_db/
├── schema.prisma                          # datasource + generator blocks
├── bookstore_db.config.ts                 # Prisma 7 config (URL via env)
├── package.json, tsconfig.json            # runnable project scaffold
├── .env.example, .gitignore, README.md    # copy .env.example to .env (git-ignored, never generated)
├── bookstore_v1/bookstore.postgres.prisma # models & enums, one file per source proto
└── inventory/inventory.postgres.prisma    # (a second file, merged datasource)

generated/gorm/bookstore_db/bookstorev1/models.go        # package = folder name
generated/gorm/bookstore_db/bookstorev1/author_store.go  # typed CRUD store (stores opt)
generated/gorm/bookstore_db/bookstorev1/filters.go       # AIP-160/132 filter & sort specs (filters opt)
generated/gorm/bookstore_db/bookstorev1/protobuf.go      # proto ↔ model converters (converters opt)
generated/gorm/gormx/gormx.go                       # shared runtime: ListOptions, Store[M], engine (stores opt)
generated/gorm/filterx/                             # shared filter/order/list engines: SQL + Hasura (filters opt)
generated/gorm/bookstore_db/migrate.go              # factory Registry + EnsureSchemas + Instrument (needs go_module)
generated/gorm/bookstore_db/README.md               # ER diagram + model reference
generated/sql/bookstore_db/migrate.sql              # whole DB, one transactional file
generated/sql/bookstore_db/bookstore_v1.postgres.sql
generated/sql/bookstore_db/README.md

The Prisma output is a project you can run immediately:

cd generated/prisma/bookstore_db
cp .env.example .env        # then set BOOKSTORE_DB_DATABASE_URL
npm install
npm run prisma:generate

.env is never generated — the plugin emits only .env.example. A protoc plugin overwrites every file it writes and cannot read what is already on disk, so generating .env would destroy your edited credentials on the next buf generate. .env stays git-ignored and yours; .env.example is the committed template, regenerated each run and always carrying a placeholder rather than the datasource url from your proto.

The gorm target emits a migrate.go factory registry (when you pass the go_module opt, see Plugin options). Attach it in your application — one call migrates every model across every schema, and you can register your own models alongside the generated ones:

import bookstoredb "github.com/me/gen/bookstore_db"

if err := bookstoredb.Default.EnsureSchemas(db); err != nil { // create Postgres schemas first
    log.Fatal(err)
}
if err := bookstoredb.Default.Migrate(db); err != nil { // db is your *gorm.DB
    log.Fatal(err)
}
bookstoredb.Default.Register(&MyModel{})  // add your own to the same registry
GORM stores and tracing

Two opt-in extras layer onto the gorm target's runtime:

Stores (stores opt, also needs go_module) generate a typed CRUD store per resource — one small <model>_store.go file each — plus a shared gormx runtime package they all import, so you don't hand-write the boilerplate. Each store is derived entirely from the resource's schema (PK, unique columns, foreign keys):

store := bookstorev1.NewAuthorStore(db)

a, err := store.GetByID(ctx, id)                 // primary key
a, err = store.GetByName(ctx, "authors/rowling") // a UNIQUE column → GetBy<Col>
list, err := store.List(ctx, gormx.ListOptions{Limit: 20, OrderBy: "display_name"})
n, err := store.Count(ctx, gormx.ListOptions{})
books, err := bookstorev1.NewBookStore(db).
    ListByAuthorID(ctx, a.ID, gormx.ListOptions{}) // a foreign key → ListBy<FK>

Every store exposes Create, GetByID, List, Count, Update, DeleteByID, plus GetBy<Col> finders for unique columns (including single-column unique indexes) and ListBy<FK> finders for foreign keys. The shared gormx package holds ListOptions (Limit / Offset / OrderBy / Where + Args), a generic Store[M] interface every store satisfies, and a GenericStore[M] engine that runs CRUD for any model — so one engine can drive every entity. Enabling stores adds a gorm.io/gorm dependency to the models package.

Telemetry (telemetry opt, off by default) folds first-party opentelementry observability into the generated tree — spans, per-operation metrics, and trace-correlated logs — with no third-party otel library involved. Every generated store gains a Telemetry gormx.Telemetry field (nil is a no-op, so existing callers keep compiling) and a WithTelemetry chainer; wire the generated adapter once and every instrumented store, and the migration Registry, observe through it:

o, err := opentelementry.New().
    WithService("bookstore-api", "1.0.0").
    WithOTLP("localhost", 4317).
    WithTracing().
    Build()
if err != nil {
    log.Fatal(err)
}
defer o.Close()

if err := bookstoredb.Default.Instrument(db, o); err != nil { // SQL-level spans/metrics
    log.Fatal(err)
}
store := bookstorev1.NewAuthorStore(db).WithTelemetry(telemetry.New(o)) // per-store spans/metrics

Mark low-cardinality fields as span labels with (telemetry.v1.telemetry_field) (e.g. a book's genre); tune a table's span prefix or opt it out entirely with (telemetry.v1.telemetry). Metrics stay scoped to the database operation itself (table, op, status, duration) by design — domain-level metrics belong to the application, not the ORM. It needs go_module (the adapter package lives alongside the aggregator) and adds the github.com/the-protobuf-project/opentelementry/opentelementry-go dependency. Tune the generated default — including metrics/logs-off — via store.yaml telemetry:.

AIP-160 filters and list engines (SQL + Hasura)

The filters opt (gorm target, needs go_module) generates the complete list-query surface for every resource — AIP-160 filter, AIP-132 order_by, and pagination — as specs (data) + engines (code):

  • <schema>/filters.go — a filterx.Spec per resource: the filterable fields with type-derived operator kinds (text, enum, date, timestamp, int, bool, tags, reference), the free-text Search columns, and the order_by allowlist. Tune what each field exposes with (store.v1.query).
  • filterx/ — one shared, chainable SDK package holding the engines the specs drive. filterx.Gorm[M] renders parsed conditions to SQL on a *gorm.DB; filterx.Hasura[M] renders the same filter strings to Hasura DDN GraphQL BoolExp predicates — so a gRPC/REST list endpoint and a Hasura-backed GraphQL API accept identical filters by construction.
// SQL — pass a *gorm.DB already parent-scoped and preloaded; the engine
// only adds filter / order / pagination.
rows, next, err := filterx.Gorm[bookstorev1.Author](bookstorev1.AuthorFilterSpec).
    List(ctx, db, in) // in: filterx.ListInput{PageSize, PageToken, OrderBy, Filter}

// GraphQL — same spec, same filter strings, over a Hasura DDN query handler.
rows, next, err = filterx.Hasura[bookstorev1.Author](bookstorev1.AuthorFilterSpec, queryHandler).
    Scope(scopePredicates...). // fixed predicates ANDed into every query
    List(ctx, in)

Both engines share one set of semantics, so the backends always agree: enum values normalize (ROOM and UNIT_TYPE_ROOM both match), dates and numbers validate before any query runs, : does a case-insensitive contains with ILIKE escaping, bareword terms (filter: "beach resort") match the search columns, resource-reference fields compare the bare id segment (operator = "operators/op1" and = "op1" are equivalent), and results paginate limit+1 with an opaque page token. Invalid filter/order input is rejected with filterx.ErrInvalid — translate it to your invalid-argument error (e.g. gRPC InvalidArgument) with errors.Is.

Distributed tracing into Hasura's own engine spans: Hasura DDN's engine (ddn-engine) emits its own OTEL spans (parse/validate/plan/execute) and accepts a standard W3C traceparent header on the incoming request — set TracePropagator: propagation.TraceContext{} on the network.ConnectionOptions your generated GraphQL client connects with, and every query issued through a context carrying an active span (e.g. inside tel.Span(ctx, ...) from Telemetry, or any go.opentelemetry.io/otel span) has Hasura's engine spans nest as children of yours in the same trace — verified against a live ddn-engine + Tempo. No store codegen involved: this is a runtime-go/network capability the generated queryHandler already threads ctx through to, the same way Headers carries auth tokens today.

Every engine is tunable through its chainable options: Override installs a custom handler for one filter field (the escape hatch for derived predicates the schema can't express), and Observe plugs an Observer in for query spans and rejected-input debug events. The telemetry opt additionally emits a ready-made opentelementry observer adapter (filterx.OpentelementryObserver). The Hasura engine adds a dependency on github.com/the-protobuf-project/runtime-go (its graphql package).

Proto ↔ model converters

The converters opt (gorm target) emits a protobuf.go per schema package with a mapper pair per resource — <Model>ToProto / <Model>FromProto — plus per-enum value mappers. The converters cover the mechanical field mass: scalars, enums, temporals, arrays, JSON, optional fields (pointer-wrapped where the model is a pointer; bytes stays []byte), and belongs-to value objects on the read side. They deliberately don't invent data or wiring the schema can't know:

  • synthesized columns (surrogate ids, audit timestamps) are never set from proto input; audit timestamps still render back out;
  • resource-reference columns are skipped in both directions — the resource-name ↔ id mapping stays with the caller;
  • relationalized sub-rows (value objects) render ToProto from their preloaded associations, but FromProto graph wiring (fresh ids, FK assignment, insert order) stays with the caller, composing each sub-row's own FromProto.

The sql target emits one transactional migrate.sql you can apply in a single shot — foreign keys are deferred to ALTER statements (so creation order never matters) and every statement is guarded (IF NOT EXISTS, CREATE OR REPLACE, a DO-block for enums), so the file is idempotent and safe to re-apply. The per-schema files remain as clean, readable reference DDL.

psql "$BOOKSTORE_DB_DATABASE_URL" -f generated/sql/bookstore_db/migrate.sql

GraphQL client SDK

The same protoc-gen-store binary also generates a typed Go GraphQL client from a live server. Where the proto flow is proto → database schema, this flow is GraphQL introspection → Go client — a second source into the same co-generation factory. There is no separate CLI: it runs as a normal target during buf generate. Add a plugin entry with target=graphql and point it at the endpoint via store.yaml:

# store.yaml
graphql:
  endpoint: https://api.example.com/graphql   # or `schema: schema.graphql` (a cached GraphQL SDL file)
  admin_secret: env:HASURA_ADMIN_SECRET       # sent under the dialect's auth header
  dialect: hasura
generate:
  - target: graphql
    go_module: github.com/me/app/gql          # import path of the emitted package
    # package: appql            # defaults to base(go_module)+"ql"
    # dump_schema: true         # also write <package>/schema.json
# buf.gen.yaml — the endpoint/output details live in store.yaml; the entry is minimal
plugins:
  - local: protoc-gen-store
    out: gen                    # buf writes the client tree here
    opt: [target=graphql, config=store.yaml]

buf generate then introspects the endpoint and writes the client through the plugin response into out:. The output is a self-contained library — typed row models, a fluent predicate DSL (Id.Eq(x), And/Or/Not), single-object create/update inputs, and one method per query/mutation/subscription, in per-domain …ql packages on a small transport runtime. Every handler satisfies the generic graphql.QueryHandler[M] / graphql.MutationHandler[…] interfaces from runtime-go — the same interface the gorm target's filterx.Hasura[M] list engine consumes — so a generated client plugs straight into the AIP-160 filter engine with zero glue.

[!NOTE] Because this runs under buf, the buf module's protos still need a Go import path (a go_package option or an M mapping) even though the graphql target ignores them — buf compiles the module before invoking any plugin.

Dialects

The engine-specific conventions (bool_exp combinators, _eq/_in comparisons, insert/update/delete verb prefixes, returning / affectedRows mutation responses, the x-hasura-admin-secret auth header, scalar mappings) live behind a pluggable dialect. The built-in hasura dialect covers the Hasura / DDN / Grafbase / Prisma-GraphQL lineage; select it (or a future engine) with --dialect or the graphql.dialect key in store.yaml. Adding another GraphQL database is a new dialect value plus a registry entry — the IR builder and renderer never hardcode a convention. CRUD/filter/aggregate detection itself is derived from introspection, not hardcoded, so unconventional schemas still generate compiling code.

Generated files are collision-proof: if a schema name would clash with the runtime graphql/runtime import identifiers, the import is deterministically aliased (gqlnet/rtnet) and its references rewritten, so the output always compiles regardless of the source schema's naming.

Annotations reference

Annotations come from two modules, and the split is the important part:

import "entity/v1/annotations.proto";  // structure — what things are named
import "store/v1/annotations.proto";     // storage   — how they are stored

entity.v1 is the neutral vocabulary. It ships from this repository as its own BSR module, so any plugin built on protokit — this one, a cache, a streams publisher, a docs generator — can depend on the vocabulary alone and generate its own stubs. The Go reader over it lives here at github.com/the-protobuf-project/store/plugin/entity. Because they all agree on the same vocabulary, they derive the same databases, schemas, tables, and columns from the same protos. That agreement is what lets those plugins compose without one of them having to import another's options.

protokit itself reads neither module: it reads AIP and nothing else, and every vocabulary reaches it through a reader the plugin registers. entity.v1 lived inside protokit as entity.v1 until v1.3.0 and moved because a vocabulary about datasources and tables does not belong in an engine that also generates contracts.

store.v1 is this plugin's own. It says how a column is physically stored and queried, which no other plugin needs to agree with.

orm.v1 was removed in v2. The single pre-split vocabulary is no longer published from this repository. If your protos still import a copy of orm/v1/annotations.proto, they keep generating — entity.CompatReader reads the structural half and emits a lint diagnostic per option — but the reader is a migration aid, not a supported vocabulary. Move each option to the module that now owns it:

Was Is now
(orm.v1.datasource) (entity.v1.datasource)
(orm.v1.table)table, skip, id, timestamps, indexes (entity.v1.table)
(orm.v1.column)column, skip (entity.v1.column)
(orm.v1.column)type, max_length, precision, scale, default_value, unique, index, on_delete, on_update (store.v1.column)
(orm.v1.query) (store.v1.query)

Field semantics and field numbers are unchanged, so the migration is a rename. MIGRATING.md has the scripted version, along with the one case a script cannot finish — a single (orm.v1.column) holding both structural and physical fields, which becomes two options in two modules.

Run with strict=lint:error once you believe you are done: any option the compat reader is still supplying becomes a hard failure, named field by field.

(entity.v1.datasource) — file level
Field Description
database Database name. Files sharing a name merge into one tree. Defaults to the last proto package segment.
schema Override the schema namespace for every table in the file.
url Connection URL. Recorded in config/DDL with any credentials redacted, and never written to a generated file verbatim; Prisma reads the real value from your .env.
provider postgres (default) or mongodb.
(entity.v1.table) — message level
Field Description
table Explicit table name. Defaults to the snake_case plural of the resource.
skip Exclude the message from all output.
indexes Composite indexes: { columns: [...], unique: bool, index: "..." }.
id ID_STRATEGY_ULID / ID_STRATEGY_UUID — synthesize a generated id PK and demote the IDENTIFIER field to UNIQUE.
timestamps Add created_at / updated_at (@updatedAt / GORM autoUpdateTime).
(entity.v1.column) — field level
Field Description
column Explicit column name (defaults to the proto field name).
skip Field exists in the proto contract but not the database.
(store.v1.column) — field level
Field Description
type Explicit SQL type (escape hatch; prefer the sizing options below).
max_length VARCHAR(n) instead of the VARCHAR(255) default — provider-neutral.
precision / scale NUMERIC(p, s).
default_value SQL default expression, written verbatim.
unique, index Single-column constraint / index.
on_delete / on_update FK referential action (CASCADE, SET_NULL, …) for a resource_reference field.
(store.v1.table) — message level
Field Description
outbox Emit a companion <table>_outbox table — id, aggregate_id, event_type, payload, created_at, published_at — so a change event can be written in the same transaction as the row it describes. The table only: draining it belongs to a streams generator. Off by default.
(store.v1.query) — field level

Tunes the field's generated list-query surface — the AIP-160 filter / AIP-132 order_by specs — separately from the physical column options:

string display_name = 2 [(store.v1.query) = { search: true }];
string state = 5 [(store.v1.query) = { filterable: false, sortable: false }];
Field Description
filterable Override the type-derived default in the generated filter spec (scalar columns are filterable by default, operators inferred from the type). Presence matters: filterable: false removes the field; unset keeps the default.
sortable Override the type-derived default in the order_by allowlist (scalar, date, timestamp, and numeric columns sort by default). Presence matters, as with filterable.
search Include the column in bareword free-text search — a filter term with no field (e.g. beach resort) matches it with a case-insensitive contains. Off by default.
(telemetry.v1.telemetry) — message level

Tunes a table's generated telemetry — takes effect only with the telemetry opt (or store.yaml telemetry.enabled). Defined in the standalone telemetry.v1 module (not store.v1 — see Telemetry is a separate module); import telemetry/v1/annotations.proto to use it:

option (telemetry.v1.telemetry) = { span_prefix: "bookstore.Book" };
option (telemetry.v1.telemetry) = { enabled: false };  // opt this table out
Field Description
enabled Override the tree-wide default (every table is instrumented when the telemetry opt is on). Presence matters: enabled: false strips this table's store instrumentation; unset keeps the default.
span_prefix Override the generated span-name prefix. Defaults to <schema>.<Model>, e.g. bookstore_v1.Book → span bookstore_v1.Book/Create.
metrics Override per-table op-metric recording (defaults to store.yaml telemetry.metrics, true). Spans are unaffected.
(telemetry.v1.telemetry_field) — field level

Marks a field as a span attribute on traced writes (also from telemetry.v1):

string genre = 7 [(telemetry.v1.telemetry_field) = { label: true }];
string state = 5 [(telemetry.v1.telemetry_field) = { label: true, label_key: "book.state" }];
Field Description
label Include this field as a span attribute (an opentelementry:"trace:<name>" struct tag on the generated model field). Safe for any cardinality — spans absorb per-row values.
label_key Override the attribute name. Defaults to <model_snake>.<column>, e.g. book.genre.

Configuration — store.yaml

store.yaml is the layout config: it maps proto packages to databases and schemas without per-file annotations — the way to split a multi-service monorepo into the intended database boundaries from one central file. It's entirely optional; without it, every package falls back to the package-path defaults.

Pass it with the config plugin option:

# buf.gen.yaml
plugins:
  - local: protoc-gen-store
    out: generated/sql
    opt:
      - target=sql
      - config=store.yaml   # path to your layout config
Anatomy

A complete config showing every key:

# top-level keys
strip_version: true           # flatten the API version out of derived schema names
dedupe_schema_table: true     # strip a redundant schema word from stuttering table names

# gorm first-party opentelementry instrumentation (gorm target; see the telemetry plugin opt)
telemetry:
  enabled: true                # override the telemetry opt's master switch
  metrics: false                # spans + logs only — drop the per-operation ops counter/histogram
  logs: false                   # spans + metrics only — drop trace-correlated error logging

# datasource rules (first match wins)
datasources:
  - match: "fleet.**"         # dotted package glob; trailing ** matches any suffix
    database: fleet
    schema_depth: 3           # first 3 package segments → fleet_tracking_device

  - match: "store.apps.**"
    database: users
    schema: "{leaf}_app"      # leaf package segment (version dropped) → calendar_app
    strip_version: false      # per-rule override of the top-level default
Top-level keys
Key Type Description
datasources list Ordered list of match rules. The first rule whose match matches a package wins.
strip_version bool Drop a trailing API version from derived schema names — bookstore.v1 → schema bookstore instead of bookstore_v1. Applies to resource-type-derived and config-derived schema names, never to an explicit (entity.v1.datasource).schema annotation. A per-rule strip_version overrides this default.
dedupe_schema_table bool Rename a table whose name would stutter with its schema in a schema-qualified identifier (booking schema + bookings table → bookingBookings in tools that join schema+table, e.g. Hasura). The redundant leading schema word is stripped; for the schema's primary table — where stripping leaves nothing — the table is renamed to a generic word (resource, then entity, …). Only the generated table name changes; proto/model names are untouched.
telemetry map gorm only. Tune the first-party opentelementry instrumentation (see the telemetry plugin opt). enabled (bool) overrides the opt's master switch. metrics (bool, default true) — set false to drop the per-operation ops counter/duration histogram tree-wide; narrow further per table with (telemetry.v1.telemetry).metrics. logs (bool, default true) — set false to drop the telemetry adapter's trace-correlated error logging.
graphql map Configures the GraphQL source: endpoint xor schema (a cached GraphQL SDL .graphql file), admin_secret (env:VAR or literal), headers (Key: Value list), dialect (default hasura), max_depth, scalars (Name=GoType list). Read by the target=graphql plugin entry.
generate list Per-target settings, keyed by target: go_module, package, runtime_module, dump_schema for the graphql target; the gorm knobs are set as plugin opts instead. buf owns each entry's output dir (out:), so it isn't set here.

Config is validated on load: unknown keys are rejected (strict decode), and each generate entry is checked for a known target / source / lang and, for the graphql target, a graphql source and a top-level graphql: block; the graphql block requires exactly one of endpoint/schema and a registered dialect. Every problem is reported at once with its key path.

Datasource rules

Each entry in datasources assigns every proto package matching match to a database and schema.

Key Type Description
match string Dotted glob over the package. ** (trailing) matches any remaining segments; * matches exactly one segment; everything else matches literally. e.g. fleet.**, store.apps.*, shop.cart.v1.
database string Database the matched packages map to. Packages routed to the same database merge into one schema tree.
schema string Literal schema name, or a template using {leaf} — the last package segment with a trailing API version dropped (store.apps.calendar.v1calendar). Takes precedence over schema_depth.
schema_depth int When schema is empty: join the first N package segments with _ to form the schema name (fleet.tracking.device at depth 3 → fleet_tracking_device).
strip_version bool Per-rule override of the top-level strip_version. Omit to inherit the global setting; set true/false to force it on/off for this rule.

[!NOTE] Within a rule, schema naming is decided in order: an explicit schema template wins; otherwise schema_depth applies; otherwise the schema stays resource-type-derived (and is then version-stripped per strip_version).

Precedence

When more than one source could name the database or schema, the most specific wins:

flowchart LR
    A["(entity.v1.datasource)<br/>per-file annotation"] -->|wins over| B["store.yaml<br/>matched rule"]
    B -->|wins over| C["package-path<br/>default"]

So you can set sane monorepo-wide defaults in store.yaml and still override a single file inline when it needs to live somewhere unusual.

Worked examples

Split two services into separate databases:

datasources:
  - match: "fleet.**"
    database: fleet
    schema_depth: 3        # fleet_tracking_device
  - match: "store.apps.**"
    database: users
    schema: "{leaf}_app"   # calendar_app

Flatten versions across one database:

strip_version: true        # acme.billing.v1 → schema "acme_billing"
datasources:
  - match: "acme.**"
    database: billing_db

Merge two packages into one database (their same-named models then collide, which store resolves per-target — see Determinism & migrations):

datasources:
  - match: "shop.cart.**"
    database: commerce
  - match: "shop.order.**"
    database: commerce

Plugin options

Passed via opt: in buf.gen.yaml.

Option Description
target What to emit: prisma | gorm | sql | graphql. Required. (graphql reads its endpoint from store.yaml's graphql block — see GraphQL client SDK.)
go_module gorm only. Go import path of the output directory (e.g. github.com/me/gen). Enables the migrate.go factory registry, whose package imports each per-schema models package. Omit it and the per-schema model packages still generate, just without the aggregator.
stores gorm only. Also emit a typed CRUD store per resource — one <model>_store.go file each (see GORM stores). Off by default; turning it on adds a gorm.io/gorm dependency to each models package.
filters gorm only. Emit AIP-160 filter / AIP-132 order_by specs per schema (filters.go) plus the shared filterx engine package serving both SQL (GORM) and Hasura DDN GraphQL (see filters and list engines). Off by default; requires go_module. The Hasura engine adds a github.com/the-protobuf-project/runtime-go dependency. With telemetry, also emits an opentelementry Observer adapter (filterx.OpentelementryObserver) for the list engines' spans and debug events.
converters gorm only. Emit protobuf.go proto ↔ model converters per schema — <Model>ToProto / <Model>FromProto plus enum value mappers (see converters). Off by default.
telemetry gorm only. Fold first-party opentelementry instrumentation into the generated output — instrumented stores (with stores), a telemetry adapter/plugin package, a filterx observer (with filters), and Registry.Instrument (see Telemetry). Off by default; takes effect with go_module, and adds the github.com/the-protobuf-project/opentelementry/opentelementry-go dependency. Tune it further via store.yaml telemetry: and telemetry.v1's (telemetry.v1.telemetry)/(telemetry.v1.telemetry_field) annotations.
strict Per-rule severity for schema problems. "" (default) warns on everything; true makes every rule a hard error; a spec like ref:error,collision:warn,index:error,lint:warn sets severity per rule. Rules: ref (unresolved/dropped references), collision (global name qualification), index (index names an unknown column), lint (validate-on-generate advisories).
config Path to a store.yaml layout config.
M<proto>=<import> Go import-path mapping for a proto file, required when protos omit option go_package.

[!NOTE] Migration: the otel and pulse opts, and store.yaml's otel: block, are removed — an unknown opt:/yaml key now fails the build loudly rather than silently doing nothing. Replace them with telemetry and store.yaml telemetry:. Registry.Instrument now takes an *opentelementry.Opentelementry handle (Instrument(db, o), not Instrument(db, opts ...tracing.Option)), and filterx.PulseObserver is now filterx.OpentelementryObserver.

Defaults applied automatically

store bakes in the conventions a hand-written production schema uses, so the common case needs no annotations. Each is overridable.

Default Behavior Override
Surrogate keys Every resource gets a ULID id primary key; the AIP name becomes @unique. (entity.v1.table).id
AIP system fields create_time/update_time → auto-managed NOT NULL timestamps; delete_time → nullable indexed soft-delete marker; uidUNIQUE. (AIP-148/164) rename the field
Parent materialization Each parent segment of the AIP resource pattern (users/{user}/…) becomes a FK column (user_idUser) with onDelete: Cascade. declare the field explicitly
FK indexing Every foreign-key column gets a single-column @@index (Postgres does not auto-index FKs). already indexed columns are skipped
Enum hygiene The AIP *_UNSPECIFIED = 0 sentinel is dropped; a required enum column defaults to its first value. (store.v1.column).default_value
oneof integrity A oneof adds a <oneof>_case discriminator enum recording which member is set.
Soft FK A resource_reference to a model outside the generation set is kept as an indexed scalar column with a TODO note, not dropped. provide the referenced resource
Relationalized nesting Every message-typed field becomes its own child table with a primary key + foreign key — never an opaque JSONB blob — so the structure stays queryable. This covers user-defined nested messages and imported value types (google.type.Money, PostalAddress, a third-party proto), read straight from the descriptor set protoc already supplies — no source or network fetch. Required links cascade on delete, optional links null. (map fields and the freeform google.protobuf wrappers — Struct, Any, Value, ListValue, Empty — stay JSONB; well-known scalar types like Timestamp stay single columns.) (store.v1.column).on_delete

Determinism & migrations

Generation is deterministic: re-running on unchanged protos produces byte-identical output (enforced by golden tests), so a regenerate → prisma migrate diff is a no-op when nothing changed. When two schemas in one database share a model or enum name, only Prisma qualifies the colliding names (its models occupy one global namespace) — and it qualifies all participants, so adding a new package cannot silently rename an existing model and force a destructive migration. The schema-namespaced targets (SQL, GORM) keep the bare name, since the schema or Go package already disambiguates it. Recommended flow: regenerate, review the diff, then migrate diff / migrate dev.

Type mapping

The IR stores a neutral, target-agnostic type per column; store projects it onto a canonical PostgreSQL type, then onto each backend's own type system. Highlights:

Proto PostgreSQL Prisma Go
string VARCHAR(255) String string
int32 INTEGER Int int32
int64 BIGINT BigInt int64
uint64 NUMERIC(20,0) Decimal string
bool BOOLEAN Boolean bool
bytes BYTEA Bytes []byte
enum a CREATE TYPE enum enum typed string consts + CHECK constraint
Timestamp TIMESTAMPTZ DateTime time.Time
Duration INTERVAL String string
double / float DOUBLE PRECISION / REAL Float float64 / float32
map / freeform msg (Struct, Any) JSONB Json json.RawMessage
nested / imported value msg child table (PK + FK) relation relation struct
repeated scalar T[] T[] []T

Unsigned 32/64-bit kinds widen one step (uint32BIGINT) so the full range fits. Well-known types with a clean single-column form — Timestamp, Duration, the wrappers, google.type.Date / LatLng / Decimal — map to a column; structured value types (google.type.Money, PostalAddress, …) relationalize into a child table instead (see Relationalized nesting). Nullable columns become pointer (*T) / optional (T?) types.

Examples

The examples/ directory is a complete, generated demo — a bookstore domain rendered to store's three database targets:

examples/proto/bookstore/v1/   # annotated source protos
examples/generated/prisma/     # ─┐
examples/generated/gorm/       #  ├─ regenerated output, one tree per target
examples/generated/sql/        # ─┘

Regenerate it with:

buf generate --template buf.gen.example.yaml

Building from source

git clone https://github.com/the-protobuf-project/store
cd store
go build ./plugin/cmd/protoc-gen-store   # the plugin binary
go test ./...                              # golden + unit tests
buf lint                                   # proto linting

Releases & versioning

  • Releases are cut by pushing a vX.Y.Z tag; GoReleaser builds cross-platform archives (linux/darwin/windows · amd64/arm64), publishes a GitHub Release with a categorized changelog, and updates the Homebrew tap.
  • Versioning follows semantic version tags. While the project is in early development (v0.x), minor releases may include breaking changes to the API or generated output — pin an exact tag in CI and review migration diffs.
  • The annotation module is published to the Buf Schema Registry under entity.v1 (structure) and store.v1 (storage); option field numbers live in the 5000099999 range reserved for non-Google custom options.

See the Releases page for binaries and changelogs, and SECUIRTY.MD for the security policy.

License

Licensed under the Apache License, Version 2.0.

Directories

Path Synopsis
plugin
cmd/protoc-gen-store command
Command protoc-gen-store is a protoc plugin that reads proto descriptors annotated with google.api.*, entity.v1.*, and store.v1.* options, then generates database schema artifacts for the requested backend: gorm, sql, or prisma.
Command protoc-gen-store is a protoc plugin that reads proto descriptors annotated with google.api.*, entity.v1.*, and store.v1.* options, then generates database schema artifacts for the requested backend: gorm, sql, or prisma.
entity
Package entity is the reader for entity.v1, the neutral schema vocabulary that decides what things are *named* — databases, schemas, tables, columns — as distinct from how any one generator stores them.
Package entity is the reader for entity.v1, the neutral schema vocabulary that decides what things are *named* — databases, schemas, tables, columns — as distinct from how any one generator stores them.
factory/config
Package config loads and validates store.yaml — the factory's single source of truth.
Package config loads and validates store.yaml — the factory's single source of truth.
factory/coreir
Package coreir is the factory's shared intermediate model.
Package coreir is the factory's shared intermediate model.
factory/facets
Package facets is the one place that knows how this plugin's annotation vocabularies are stored on protokit's IR, and the only way its targets read them back.
Package facets is the one place that knows how this plugin's annotation vocabularies are stored on protokit's IR, and the only way its targets read them back.
factory/provenance
Package provenance renders the generated-file banner with the full set of modules that decided the output.
Package provenance renders the generated-file banner with the full set of modules that decided the output.
factory/source/graphql
Package graphql is the factory Source that reads a GraphQL server: it introspects a live endpoint (or reads a cached GraphQL SDL schema, .graphql), then builds the GraphQL IR under the configured dialect.
Package graphql is the factory Source that reads a GraphQL server: it introspects a live endpoint (or reads a cached GraphQL SDL schema, .graphql), then builds the GraphQL IR under the configured dialect.
factory/source/proto
Package proto is the factory Source that reads proto descriptors.
Package proto is the factory Source that reads proto descriptors.
factory/source/proto/backend
Package backend is this plugin's bridge to protokit: the reader that brings store.v1 into protokit's IR, and the layout policy resolved from store.yaml.
Package backend is this plugin's bridge to protokit: the reader that brings store.v1 into protokit's IR, and the layout policy resolved from store.yaml.
factory/target/database
Package database adapts protokit's proto-bound schema.Target (gorm, sql, prisma) to the factory's source-agnostic factory.Target.
Package database adapts protokit's proto-bound schema.Target (gorm, sql, prisma) to the factory's source-agnostic factory.Target.
factory/target/gorm
Package gorm generates production-ready Go structs with GORM struct tags.
Package gorm generates production-ready Go structs with GORM struct tags.
factory/target/graphql
Package graphql is the "graphql" factory Target: it renders a typed GraphQL client from the Model's GraphQL facet.
Package graphql is the "graphql" factory Target: it renders a typed GraphQL client from the Model's GraphQL facet.
factory/target/graphql/golang
Package golang renders a Go client from the IR using an interface/handler architecture grouped by domain.
Package golang renders a Go client from the IR using an interface/handler architecture grouped by domain.
factory/target/graphql/golang/selection
Package selection renders Go model struct bodies from IR objects.
Package selection renders Go model struct bodies from IR objects.
factory/target/graphql/golang/typemap
Package typemap maps GraphQL types to Go types for the generated client.
Package typemap maps GraphQL types to Go types for the generated client.
factory/target/prisma
Package prisma generates a multi-file Prisma schema tree from the store IR, replicating the hand-written layout this repository uses:
Package prisma generates a multi-file Prisma schema tree from the store IR, replicating the hand-written layout this repository uses:
factory/target/repository
Package repository generates provider-agnostic, proto-facing repository layers from the same proto-derived schema the database targets render: one CRUD interface per resource (speaking proto messages and AIP conventions — resource names, page tokens, field masks, etags) plus adapters composing the generated gorm output (models, stores, converters, filterx) and, when a client module is configured, the generated GraphQL client.
Package repository generates provider-agnostic, proto-facing repository layers from the same proto-derived schema the database targets render: one CRUD interface per resource (speaking proto messages and AIP conventions — resource names, page tokens, field masks, etags) plus adapters composing the generated gorm output (models, stores, converters, filterx) and, when a client module is configured, the generated GraphQL client.
factory/target/sql
Package sql generates PostgreSQL DDL from the store IR.
Package sql generates PostgreSQL DDL from the store IR.
factory/target/types
Package types is store's projection of the neutral schema.FieldType onto the canonical PostgreSQL type the gorm/sql/prisma targets render from.
Package types is store's projection of the neutral schema.FieldType onto the canonical PostgreSQL type the gorm/sql/prisma targets render from.
factory/wire
Package wire is the factory's composition root: it assembles the registered sources and targets into one Registry.
Package wire is the factory's composition root: it assembles the registered sources and targets into one Registry.
Package telemetry owns everything this repository knows about telemetry.v1: reading the annotations, resolving what each table's instrumentation should be, and rendering the generated adapter that talks to the SDK.
Package telemetry owns everything this repository knows about telemetry.v1: reading the annotations, resolving what each table's instrumentation should be, and rendering the generated adapter that talks to the SDK.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL