flexitype

README

flexitype

Soft types and attributes for Go: define entity types, typed and constrained attributes, and attribute dependencies at runtime — then attach validated values to your own domain objects. Inspired by PLM-class flexible attribute systems, built as a production-grade DDD Go service.

New here? The getting-started guide takes you from nothing to a validated, queryable value with a cascading dependency in ten minutes.

Versioning. Releases are tagged (vX.Y.Z) with a CHANGELOG; pin a version rather than tracking main. SemVer applies from 1.0 — see API stability. Embed with go get github.com/zkrebbekx/flexitype@vX.Y.Z, or grab a standalone binary from Releases.

Embedding it? docs/embedding.md collects the library-mode contracts — request-scoped interactors, the tenant/actor/access stamp, migrations, background loops — that previously lived only in scattered package comments.

Deploying it? deploy/kubernetes/ has working manifests for the API and worker tiers, with probes, scrape config and the two alerts worth having.

Sizing it? docs/scale.md publishes measured numbers and hot-table maintenance guidance. Verifying a release? Binaries carry cosign signatures, SLSA provenance and an SBOM — see docs/releases.md.

Runs two ways from one codebase:

• Embedded library — wire it into your service over your own *sqlx.DB
• Standalone service — a single binary with a versioned REST API, service-account auth, OpenTelemetry and health endpoints

▶ Try the playground — the full service (usecases, REST API, FQL, GraphQL, search index, schema templates) compiled to WebAssembly, running the admin console entirely in your browser. No backend; data resets on reload.

Features

• Soft types: TypeDefinition → AttributeDefinition → AttributeValue, anchored to your entities via an opaque entity_id
• 14 data types: bool, string, integer, float, decimal (arbitrary precision), date, time, datetime, enum, url, email, json, media (files backed by a blob store), quantity (a magnitude in a unit family)
• Constraints: min/max length, min/max value, RE2 pattern, one-of, media (allowed MIME types + max size), plus required / multi-valued / unique attribute flags
• Localized & scoped values: mark an attribute localizable and/or scopable and hold a distinct value per locale and channel; single-valued uniqueness and FQL filtering apply per scope, and a query can pin one (GET /query?locale=fr&channel=web)
• Computed attributes: derive a value from a formula over other attributes ((price - cost) / price) — materialized by an event subscriber so the result is an ordinary, FQL-queryable value that stays in sync. Aggregates fold a multi-valued attribute: sum, count, avg, min, max (sum(line_totals)). A bare name must resolve to one value, so a multi-valued attribute has to be aggregated explicitly rather than collapsed to an arbitrary member
• Units of measure: a quantity attribute pins a tenant unit family (mass, length, …) with per-unit conversion factors; values normalize to a base unit so weight > 5.5 kg compares 6000 g correctly, and min/max constraints normalize too. /api/v1/unit-families
• Media attributes: upload files against a media attribute through a pluggable blob store (local disk or in-memory; S3/MinIO drop-in), with MIME and size limits and blob GC on archive. POST /api/v1/entities/{type}/{entity}/attributes/{attr}/media
• Attribute dependencies: cascading picklists and conditional validation — when a source attribute matches conditions (equals / in / range / pattern / dynamic time), the target's allowed values narrow, constraints tighten or required flips; resolve the effective schema per entity for building UIs
• Dynamic values: now / today / relative-time defaults and conditions
• Domain events: aggregates return []events.Event; a dispatcher fans a stable JSON envelope out to your infrastructure — pub/sub brokers, HMAC-signed webhooks, or plain funcs
• Activity log: every change audited with JSON before/after descriptors, written in the same transaction as the change
• Data erasure: an admin-scoped, audited, irreversible hard delete of one entity or a tenant's entity data — the right-to-erasure primitive on top of everyday soft deletes. See docs/erasure.md
• Dataloaders throughout the repositories: point lookups batch into ANY() queries, identical filter+page queries deduplicate, per-parent pagination collapses into one windowed query
• Type inheritance: single-inheritance hierarchies (MountainBike extends Bike extends Product) — subtypes inherit every attribute, constraint and dependency; no shadowing anywhere in a hierarchy; values anchor to the entity's declared type; uniqueness applies hierarchy-wide; dependencies and relationships work across levels
• Relationships between types: user-defined relationship types — directed (a parent side and a child side, optional display role labels like "assembly"/"component", per-link version binding: track the latest type version or pin one) or symmetric (unordered peers such as compatible_with; the pair is stored canonically so A↔B can never duplicate as B↔A). Both carry their own attributes and constraints (the full attribute machinery applies to links), support definition inheritance, and enforce optional cardinality bounds (min/max children per parent and parents per child).
• FQL, a schema-aware query language: query entities by attribute values and across relationships — category = "bike" and (min(price) >= 500 or "sale" in tags) and child(supplied_by) { link.lead_time_days <= 14 }. Comparisons, in, range, has, length, min/max/count, case-insensitive string matching, and/or/not with parentheses, type isa hierarchy matching; child()/parent() traverse directed relationships, linked() matches either end (the only traversal for symmetric ones). Names bind against the (inherited) schema with positioned errors; archived types, attributes and entities are invisible. See docs/design/query-language.md.
• GraphQL read API: a read-only /api/v1/graphql whose schema is generated from your live type definitions — each type an object, each attribute a field, each relationship a nested Relay connection (edges/node/cursor, pageInfo, on-demand totalCount), with FQL exposed as a filter argument. Relationship fields resolve through the dataloaders (no N+1), the schema regenerates on definition events, and introspection reflects only the caller's readable types. Opt into Apollo Federation and it also serves _service { sdl }, _entities(representations:) and @key(fields: "entityId") on every type, so a gateway resolves your dynamic attributes onto an entity another subgraph owns (FLEXITYPE_FEATURE_GRAPHQL_FEDERATION=true, or flexitype.WithGraphQLFederation()).
• Faceted grid & saved views: project chosen attributes as columns (/entities/{type}/grid), get value counts over the current result set (/entities/{type}/facets), and persist a type + query + columns as a named saved view (/api/v1/saved-views).
• CSV import & export: bulk-load entities from tabular data with column mapping, a dry-run validation report, and best-effort or all-or-nothing modes; export honours the active FQL query. POST|GET /api/v1/entities/{type}/import|export
• Duplicate detection: per-type match rules (exact, case-insensitive or trigram with a threshold) produce scored candidate pairs; dismissals stick. Scoring runs in Go so both backends agree. /type-definitions/{id}/match-rules
• Completeness scoring: score an entity (or a whole type) against its effective, dependency-adjusted required schema. /entities/{type}/{entity}/completeness
• Entity revisions: capture immutable point-in-time value snapshots, list and diff them, read an entity as of a timestamp, and restore a revision (which replays as normal writes, so events and activity fire). /entities/{type}/{entity}/revisions, /revisions/{id}
• Change management: stage a batch of value edits as a change-set (draft → in-review → approved → published), preview the result against live data without touching it, require a distinct approver, and publish now or on a schedule. /api/v1/changesets A publish claims the set (state publishing) before it applies the mutations. If the publish never finishes — a client timeout, a pod eviction — the claim goes stale after 15 minutes, and the set becomes publishable again: the scheduler retries it, and so does an explicit publish. The mutations are declarative, so a retry reaches the same state whether or not the stranded publish committed its values.
• Field-level access control: per-attribute read/write permissions on a service account gate the write path, value reads, effective-attributes and the FQL binder — an unreadable attribute is invisible rather than leaked.
• Schema templates & type cloning: bootstrap a tenant from curated, in-repo starter schemas (/api/v1/schema/templates), or clone an existing type's attributes, constraints and dependencies as a fresh root (POST /type-definitions/{id}/clone). Both reuse the portable, name-keyed schema bundle from import/export.
• Keyset cursor pagination: every list pages with an opaque keyset cursor (not offset), so pages stay stable under concurrent inserts and deletes — no skipped or duplicated rows. The total is computed only when asked for (?total=true, or the GraphQL totalCount field).
• Transactional outbox (optional): event envelopes persist in the same transaction as the change and a relay dispatches them with retries — at-least-once delivery for every hook (webhooks, pub/sub, the search indexer). FLEXITYPE_OUTBOX=true or flexitype.WithOutbox() + Service.RunOutboxRelay.
• Event delivery for other services (with the outbox): managed webhook subscriptions (/api/v1/webhook-subscriptions) with signed deliveries, exponential backoff, dead-lettering and redrive; plus a cursor-paged events feed (/api/v1/events), an SSE live tail and named compare-and-swap cursors so replicated consumers read as one. Safe with any number of flexitype replicas. Design: docs/design/event-delivery.md.
• Google Cloud Pub/Sub publisher (optional): every event as one Pub/Sub message with filterable attributes and optional per-aggregate ordering keys — the preferred integration when consumers live on GCP. FLEXITYPE_PUBSUB_PROJECT (+ _TOPIC, _ORDERING) standalone, or the infrastructure/gcppubsub handler when embedding.
• Search index (optional): an event-driven projection keeps one searchable document per entity, unlocking FQL matches("free text") and POST /api/v1/search/reindex; trigram indexes accelerate contains/icontains everywhere. FLEXITYPE_FEATURE_SEARCH_INDEX=true or flexitype.WithSearchIndex(). Design: docs/design/search-indexing.md.
• Feature toggles: search and activity history switch off per deployment (FLEXITYPE_FEATURE_SEARCH, FLEXITYPE_FEATURE_ACTIVITY, or flexitype.WithoutSearch() / WithoutActivityLog() when embedding); the console adapts automatically.
• Admin console: a built-in Vue 3 UI at / for modelling types, attributes, dependencies and relationships (including from a template or by cloning), browsing entities with a faceted, column-configurable grid and dependency-aware value editing (localized/scoped values, media upload, computed and unit-of-measure fields), running FQL and a GraphQL explorer, managing saved views, duplicates, revisions and change-sets, and auditing every change with before/after diffs
• Multi-tenant from day one; definition versioning with values pinned to the version they were validated against

Architecture

domain/          Aggregates, value objects, constraints, events, repo ports
application/     Usecases (interactors) — types, attributes, values, query,
                 relationships, schema, dedup, revision, changeset, unit,
                 computed, search, gql — plus the common factory, unit of
                 work, activity log contract, actor/tenant context
infrastructure/  PostgreSQL + in-memory repositories (dataloader-backed),
                 migrations, activity log, embedded migration runner
internal/.../http REST + GraphQL API for the standalone service
pkg/             Reusable primitives: ulid, db (Transactor + commit hooks,
                 keyset pagination), dataloader, events (dispatcher + hooks),
                 blob (media store), fql, formula, ratelimit, metrics,
                 safedial, serviceaccount, logger, config, telemetry, health
cmd/flexitype    Composition root for the standalone service (+ -wasm playground)
flexitype.go     Embedding facade
client/          First-party Go REST client (separate, stdlib-only module)

Every write flows through the unit of work: the usecase opens the transaction, repositories join it (WithTx, GetForUpdate row locks), and the common factory registers three commit handlers —

1. pre-commit → activity-log rows written inside the transaction
2. post-commit → domain events dispatched to your hooks (only after the change is durable)
3. rollback → observability hook

Embedded usage

import (
    "github.com/jmoiron/sqlx"
    _ "github.com/lib/pq"

    "github.com/zkrebbekx/flexitype"
    "github.com/zkrebbekx/flexitype/pkg/events"
)

pool, _ := sqlx.Connect("postgres", dsn)

svc := flexitype.New(pool,
    // Route events into your broker (NATS, Kafka, SNS, ...).
    flexitype.WithPublisher("nats", myNATSPublisher, nil),
    // Or deliver signed webhooks.
    flexitype.WithWebhook("billing", events.WebhookConfig{
        URL:    "https://billing.internal/hooks/flexitype",
        Secret: os.Getenv("HOOK_SECRET"),
    }),
    // Or just run a func.
    flexitype.WithHandlerFunc("cache-invalidator", func(ctx context.Context, env events.Envelope) error {
        cache.Invalidate(env.AggregateID)
        return nil
    }, events.WithEventTypes(value.EventUpdated)),
)

_ = svc.Migrate(ctx) // embedded migrations, advisory-locked, idempotent

// One interactor set per request/unit of work (fresh dataloader caches).
interactors := svc.Interactors(ctx)
product, _ := interactors.TypeDefinitions().Create(ctx, typedef.CreateInput{
    InternalName: "product",
    DisplayName:  "Product",
})

For tests and prototypes, flexitype.NewInMemory(...) takes the same options and runs the identical usecases over an in-process store — no database, no migrations. It powers the browser playground and makes a zero-dependency test double for embedding consumers.

Consumers on other stacks integrate via the standalone service's REST API and webhooks; every subscriber sees the same envelope:

{
  "id": "01J...",
  "type": "flexitype.attribute_value.updated",
  "aggregate_type": "attribute_value",
  "aggregate_id": "01J...",
  "tenant_id": "acme",
  "actor": "service_account:ci-importer",
  "occurred_at": "2026-07-11T10:00:00Z",
  "recorded_at": "2026-07-11T10:00:00.003Z",
  "schema_version": 1,
  "payload": { "old_value": "SN-100", "new_value": "SN-200", "...": "..." }
}

Webhook deliveries carry X-Flexitype-Signature (hex HMAC-SHA256 of the body); verify with events.VerifySignature.

Quickstart (Docker)

One command brings up the service (admin console embedded) and Postgres, with the transactional outbox and entity search index enabled:

docker compose up --build
# then open http://localhost:8080

The published image is available without cloning:

docker pull ghcr.io/zkrebbekx/flexitype:latest

For a realistic, end-to-end scenario — a PLM-style product catalog with inheritance, a cascading dependency, FQL and a signed webhook consumer — see examples/catalog (docker compose up + a seed script).

Standalone service

go build -o flexitype ./cmd/flexitype
FLEXITYPE_DB_HOST=localhost FLEXITYPE_DB_NAME=flexitype ./flexitype

Configuration is environment-driven (FLEXITYPE_PORT, FLEXITYPE_DB_*, FLEXITYPE_SERVICE_ACCOUNTS, FLEXITYPE_WEBHOOK_URL/_SECRET, FLEXITYPE_OUTBOX, FLEXITYPE_EVENT_RETENTION, FLEXITYPE_METRICS, FLEXITYPE_FEATURE_SEARCH_INDEX, FLEXITYPE_BLOB_DIR (media storage), FLEXITYPE_MIGRATE_ON_START, FLEXITYPE_LOG_LEVEL) — every variable is tabulated in docs/configuration.md. Tracing follows the standard OTEL_EXPORTER_OTLP_ENDPOINT. Liveness at /healthz, readiness (with a database probe) at /readyz.

Prometheus metrics are served at /metrics (unauthenticated; FLEXITYPE_METRICS=false to disable): flexitype_http_requests_total and flexitype_http_request_duration_seconds labelled by method, route pattern and status class, plus Go/process collectors. With the outbox on, scrape-time gauges flexitype_outbox_pending and flexitype_webhook_deliveries{status} report delivery depth. Embedding consumers pass a *metrics.Metrics to APIConfig.

Consuming events from another service

With FLEXITYPE_OUTBOX=true, other services subscribe over the API — no broker, no SDK. Register an endpoint:

curl -X POST /api/v1/webhook-subscriptions -d '{
  "name": "billing",
  "url": "https://billing.internal/hooks/flexitype",
  "secret": "s3cret",
  "event_types": ["flexitype.attribute_value.set", "flexitype.attribute_value.updated"]
}'

Subscription URLs must be public https endpoints: the service rejects private, loopback and link-local targets at registration and again at dial time (the delivery worker resolves the host and blocks non-public addresses, defeating DNS rebinding) — an SSRF guard. On-prem deployments whose consumers live on internal networks set FLEXITYPE_WEBHOOK_ALLOW_PRIVATE=true (or flexitype.WithWebhookAllowPrivate()) to allow http and private hosts.

Every matching envelope arrives as a signed POST, retried with exponential backoff and dead-lettered (with API redrive) after ~3 days of failures. The receiving handler needs three things:

1. Return 2xx fast; process async. Anything else retries.
2. Verify the signature — events.VerifyRequest(secrets, r.Header.Get(events.HeaderTimestamp), body, r.Header.Get(events.HeaderSignature), events.DefaultSignatureTolerance, time.Now()) checks the HMAC and rejects replays.
3. Dedupe on the envelope id (INSERT ... ON CONFLICT DO NOTHING into a processed-events table). Delivery is at-least-once by design; this one rule makes N flexitype replicas × M consumer replicas safe.

Pull consumers use the ordered feed instead: GET /api/v1/events?after= <cursor> pages expanded events (GET /api/v1/events/stream is the SSE live tail, resuming via Last-Event-ID), and named cursors (PUT /api/v1/event-cursors/{consumer} with {"position": n, "expected": m}) commit progress with compare-and-swap, so replicated consumers read as one logical consumer. Cursors older than FLEXITYPE_EVENT_RETENTION (default 7 days) get 410 CURSOR_EXPIRED — re-baseline instead of silently missing events. Full design: docs/design/event-delivery.md.

Consumers on GCP should prefer Pub/Sub: set FLEXITYPE_PUBSUB_PROJECT (topic via FLEXITYPE_PUBSUB_TOPIC, default flexitype-events; per-aggregate ordering keys via FLEXITYPE_PUBSUB_ORDERING=true) and every event publishes as one message — envelope JSON as the body, attributes (event_type, tenant_id, aggregate_id, ...) for server-side subscription filters. Pub/Sub then provides consumer groups, replay and dead-letter topics natively; dedupe on the event_id attribute as with every other lane. Embedded services register the same handler directly: flexitype.WithHandler(gcppubsub.New("gcp-pubsub", client.Publisher("flexitype-events"))).

Service accounts

Machine-to-machine auth via bearer tokens (ft_<account>_<secret>), accounts declared in a JSON file with SHA-256 secret hashes and read/write/admin scopes plus optional per-attribute field permissions; each account is pinned to a tenant. Database-backed deployments assign those permissions through roles rather than per account. (The human-identity / SSO roadmap is in docs/design/identity.md.)

[
  {
    "id": "ci",
    "name": "CI Importer",
    "tenant_id": "acme",
    "scopes": ["read", "write"],
    "secret_hash": "<hex sha256 of the secret>"
  }
]

Authentication is required by default: with no account source configured the service refuses to boot. To run without it — which serves the whole API, including the irreversible admin purge, to anonymous callers — set FLEXITYPE_DEV_INSECURE=true explicitly. That flag also permits an unencrypted database connection to a non-loopback host, which is what the compose quickstart needs.

Field permissions

A field permission names an attribute and a level: none, read or write. An account holding the admin scope, or holding no field permissions at all, reads and writes every attribute.

The permission set applies to every surface that returns an attribute value:

Surface	Behaviour for an unreadable attribute
GET /api/v1/values, ListByEntity, Get by id	the value is omitted; a Get by id returns 404
Grid, facets, CSV export, FQL	the attribute is unknown — it cannot be selected, bucketed or filtered
GraphQL, including introspection	the attribute is not a field on the type: the schema is built from the caller's readable set and cached per permission profile, so its name is not disclosed either
Revisions (Get, AsOf, Diff)	the value is omitted from the snapshot and from the diff
Activity log	the audit entry survives; its before/after values are null and the entry carries "redacted": true
Events feed	the envelope and its sequence survive; the payload's value fields are null and it carries "redacted": true
Media download	GET /api/v1/media/{key} returns 404
Duplicate detection	a match rule on the attribute cannot be created, and an existing rule cannot be scanned

Writes are symmetric: a write level is required to set or remove a value, and removing a whole entity requires write on every attribute that entity holds.

Roles

A role names a permission set once, so many accounts share it instead of each carrying a copy. An account holds role names; the effective permission is merged at authentication, so a change to a role reaches every holder as soon as the auth cache entry expires.

curl -X PUT $BASE/api/v1/roles -H "Authorization: Bearer $ADMIN" \
  -d '{"tenant_name":"acme","name":"analyst","scopes":["read"],
       "field_permissions":{"salary":"none"}}'

curl -X POST $BASE/api/v1/service-accounts -H "Authorization: Bearer $ADMIN" \
  -d '{"tenant_name":"acme","name":"jamie","roles":["analyst"]}'

Scopes union across roles, field permissions take the most permissive level any role grants, and the account's own entry wins over every role. The full rules are in docs/design/identity.md.

Embedders stamp the policy themselves with uow.WithAccess. Set uow.Access.Default to uow.PermNone to turn the permission set into an allow-list, so an attribute added later is unreadable until it is granted. Select flexitype.WithFailClosedACL() so a request that carries no policy denies everything instead of granting admin, and stamp uow.WithSystemAccess on host-owned background work that has no principal.

Runtime provisioning (multi-tenant control plane)

Instead of (or as well as) a static file, run with FLEXITYPE_PROVISIONING=true to keep tenants and service accounts in the database and manage them at runtime through the admin API — the onboarding path for the hosted, multi-tenant story:

POST/GET/PATCH  /api/v1/tenants
POST/GET        /api/v1/service-accounts        (token shown once, on create)
POST/DELETE     /api/v1/service-accounts/{id}/rotate|revoke

All of these require the admin scope. Bootstrap the first admin credential with FLEXITYPE_BOOTSTRAP_ADMIN=true — its token is logged once at startup; capture it. See docs/configuration.md for the full env-var reference and docs/getting-started.md for a first-run walkthrough.

REST API (v1)

The full contract is published as OpenAPI 3 — committed at api/openapi.yaml and served (unauthenticated) at /api/v1/openapi.json and /api/v1/openapi.yaml.

Go services get a first-party, hand-crafted client at github.com/zkrebbekx/flexitype/client — a standard-library-only module that mirrors the embedded usecase surface over the network, with keyset pagination iterators and typed errors:

c, _ := client.New("https://flexitype.internal", client.WithToken(tok))
prod, _ := c.Types().Create(ctx, client.CreateTypeInput{InternalName: "product", DisplayName: "Product"})
for row, err := range c.Query(ctx, "product", `price > 100`) { /* ... */ }
if errors.Is(err, client.ErrNotFound) { /* ... */ }

For other languages, generate a client from the OpenAPI document. Both are covered in docs/clients.md.

GET|POST   /api/v1/type-definitions            PATCH /api/v1/type-definitions/{id}
POST       /api/v1/type-definitions/{id}/archive|restore
POST       /api/v1/type-definitions/{id}/clone
GET        /api/v1/type-definitions/{id}/attributes
GET        /api/v1/type-definitions/{id}/effective-attributes
GET        /api/v1/type-definitions/{id}/children
GET        /api/v1/type-definitions/{id}/completeness
GET|POST   /api/v1/type-definitions/{id}/match-rules
GET|POST   /api/v1/attributes                  PATCH /api/v1/attributes/{id}
POST       /api/v1/attributes/{id}/archive|restore
POST       /api/v1/attributes/{id}/validate-value
GET|POST   /api/v1/values                      GET|DELETE /api/v1/values/{id}
POST       /api/v1/values/batch
GET        /api/v1/entities/{typeDef}                (list)
GET        /api/v1/entities/{typeDef}/grid|facets    (faceted grid)
POST|GET   /api/v1/entities/{typeDef}/import|export  (CSV)
GET        /api/v1/entities/{typeDef}/{entity}/values|completeness|as-of
DELETE     /api/v1/entities/{typeDef}/{entity}                 (archive, cascade)
POST       /api/v1/entities/{typeDef}/{entity}/purge           (erase, admin, hard delete)
GET        /api/v1/entities/{typeDef}/{entity}/attributes/{attr}/effective-schema
POST       /api/v1/entities/{typeDef}/{entity}/attributes/{attr}/media
GET|POST   /api/v1/entities/{typeDef}/{entity}/revisions
GET        /api/v1/revisions/{id}              GET /api/v1/revisions/{id}/diff  POST .../restore
GET        /api/v1/media/{objectKey}
GET|POST   /api/v1/dependencies                PATCH|DELETE /api/v1/dependencies/{id}
GET|POST   /api/v1/unit-families               GET|DELETE /api/v1/unit-families/{id}
GET|POST   /api/v1/saved-views                 GET|PATCH|DELETE /api/v1/saved-views/{id}
GET|POST   /api/v1/changesets/...              (submit|approve|reject|publish|mutations)
GET        /api/v1/schema/export|templates     POST /api/v1/schema/import|templates/{name}/apply
GET        /api/v1/features
GET        /api/v1/query?type=&q=&locale=&channel=&total=   POST /api/v1/query/validate
GET|POST   /api/v1/graphql                     (read-only, schema from your types)
POST       /api/v1/search/reindex
GET|POST   /api/v1/relationship-definitions    PATCH /api/v1/relationship-definitions/{id}
POST       /api/v1/relationship-definitions/{id}/archive|restore
GET        /api/v1/relationship-definitions/{id}/attribute-sets
GET|POST   /api/v1/relationships               GET|DELETE /api/v1/relationships/{id}
GET        /api/v1/entities/{typeDef}/{entity}/relationships
GET        /api/v1/match-rules/{id}/scan|dismiss                (duplicate detection)
GET        /api/v1/activity
POST       /api/v1/admin/purge                 (erase tenant entity data, admin, hard delete)
GET|POST   /api/v1/webhook-subscriptions       GET|PATCH|DELETE /api/v1/webhook-subscriptions/{id}
GET        /api/v1/webhook-subscriptions/{id}/deliveries?status=
POST       /api/v1/webhook-deliveries/{id}/redeliver
GET        /api/v1/events?after=&types=        GET /api/v1/events/stream (SSE)
GET|PUT    /api/v1/event-cursors/{consumer}

Lists paginate with ?limit= and an opaque, keyset ?cursor= (stable under concurrent writes); the total count is computed only when asked (?total=true). Bad pagination params (a non-positive limit, a malformed cursor) return 422 uniformly. Errors carry stable machine codes (VALIDATION, NOT_FOUND, CONFLICT, ARCHIVED, DEPENDENCY_VIOLATION, FORBIDDEN, RATE_LIMITED).

Example: cascading picklist

# category: enum(bike, car)      subcategory: enum(mountain, road, sedan, suv)
curl -X POST :8080/api/v1/dependencies -d '{
  "source_attribute_id": "'$CATEGORY'",
  "target_attribute_id": "'$SUBCATEGORY'",
  "conditions": [{"kind": "equals", "value": {"type": "enum", "value": "bike"}}],
  "effect": {"allowed_values": [
    {"type": "enum", "value": "mountain"},
    {"type": "enum", "value": "road"}
  ]}
}'

# With category=bike set on product-9, the UI asks what subcategory may be:
curl :8080/api/v1/entities/$TYPE/product-9/attributes/$SUBCATEGORY/effective-schema
# → {"required":false,"restricted":true,"allowed_values":["mountain","road"], ...}

Admin console

The standalone service ships a built-in admin console at / (the API stays under /api/v1). Develop it with the Go service running:

cd web && npm ci && npm run dev   # http://localhost:5173, proxies /api

The console ships only in the container image and the published release binaries. Both run npm run build before compiling. A binary built any other way — including from a plain checkout — embeds a committed stub that serves a page saying the console was not built in. The stub is what keeps go build working without a Node toolchain.

To embed it yourself: npm ci && npm run build in web/, then go build -C cmd/flexitype .. The server is its own Go module, so build it from its directory rather than from the repository root.

Upgrades

Migrations run at startup by default, each in its own transaction, serialized across replicas by an advisory lock. Index builds on the value table run CONCURRENTLY and data backfills run in bounded batches after the schema is in place, so an upgrade never holds a write-blocking lock over a whole-table scan.

Release N's migrations stay compatible with release N-1's binary, so a rolling deploy is safe and a rollback is redeploying the previous binary — not running MigrateDown, which is a development tool. A binary that finds a schema newer than itself logs a warning, so a mixed-version fleet is visible rather than inferred.

The full contract, and the rules for writing a migration that is safe against a live fleet, are in docs/upgrades.md.

Development

go build ./...   # everything compiles without a database
go test ./...    # goconvey Given/When/Then suites
go vet ./...
cd web && npm test && npm run build   # console tests + typecheck + bundle

./scripts/coverage.sh                 # statement coverage + per-package report

Coverage is measured with -coverpkg across the module, because most of it comes from cross-package tests (the root integration suites and the in-memory feature suites drive application/, domain/ and infrastructure/). Set FLEXITYPE_TEST_DSN first — the Postgres suites skip without it, which sinks the total. MIN_COVERAGE=NN enforces a floor (CI does); COVERAGE_HTML=cov.html writes a browsable report.

Storage is a single polymorphic value table with one typed, indexed column per storage class — no table-per-type explosion, uniqueness probes stay index-backed, and entity hydration is one composite-index scan.

License

MIT

Documentation

Overview

Package flexitype is the embedding facade: everything needed to run flexitype inside your own Go service — usecases, storage, migrations, domain events — wired through one constructor with hook options for your pub/sub, webhooks or plain functions. For the standalone service, see cmd/flexitype.

Index

• type APIConfig
• type DeliveryLoops
  • func AllDeliveryLoops() DeliveryLoops
• type Option
  • func WithBackgroundErrorObserver(fn func(err error)) Option
  • func WithBlobStore(s blob.Store) Option
  • func WithCleanupObserver(fn func(err error)) Option
  • func WithDeadLetterRetention(d time.Duration) Option
  • func WithDeliveryWorker(opts ...webhook.WorkerOption) Option
  • func WithDispatchObserver(fn func(ctx context.Context, err error)) Option
  • func WithEventRetention(d time.Duration) Option
  • func WithFailClosedACL() Option
  • func WithGraphQLFederation() Option
  • func WithHandler(h events.Handler, opts ...events.RegisterOption) Option
  • func WithHandlerFunc(name string, fn func(ctx context.Context, env events.Envelope) error, ...) Option
  • func WithOutbox(opts ...outbox.RelayOption) Option
  • func WithPublisher(name string, pub events.Publisher, topicFn events.TopicFunc) Option
  • func WithRollbackObserver(fn func(ctx context.Context, err error)) Option
  • func WithSearchIndex() Option
  • func WithTimeZone(loc *time.Location) Option
  • func WithWebhook(name string, cfg events.WebhookConfig, opts ...events.RegisterOption) Option
  • func WithWebhookAllowPrivate() Option
  • func WithWebhookTimeout(d time.Duration) Option
  • func WithoutActivityLog() Option
  • func WithoutSearch() Option
• type Service
  • func New(pool *sqlx.DB, opts ...Option) *Service
  • func NewInMemory(opts ...Option) *Service
  • func (s *Service) APIHandler(cfg APIConfig) http.Handler
  • func (s *Service) AdminInteractor(opts ...admin.Option) *admin.Interactor
  • func (s *Service) BootstrapAdmin(ctx context.Context, tenantName, accountName string) (string, error)
  • func (s *Service) Context(ctx context.Context) context.Context
  • func (s *Service) Dispatcher() *events.Dispatcher
  • func (s *Service) EnsureWebhookSubscription(ctx context.Context, name, url, secret string, eventTypes ...string) error
  • func (s *Service) Factory() application.Factory
  • func (s *Service) GraphQLEngine() *gql.Engine
  • func (s *Service) Interactors(ctx context.Context) *application.Interactors
  • func (s *Service) Migrate(ctx context.Context) error
  • func (s *Service) NewAPIHandler(cfg APIConfig) (http.Handler, error)
  • func (s *Service) NewAccountLookup(ttl time.Duration) serviceaccount.Authenticator
  • func (s *Service) RecomputeComputed(ctx context.Context, tenant valueobjects.TenantID) (int, error)
  • func (s *Service) ReindexSearch(ctx context.Context, tenant valueobjects.TenantID) (int, error)
  • func (s *Service) RunChangeSetScheduler(ctx context.Context, interval time.Duration)
  • func (s *Service) RunOutboxRelay(ctx context.Context, loops ...DeliveryLoops)
  • func (s *Service) SchemaDrift(ctx context.Context) ([]int, error)

Types

type APIConfig

type APIConfig struct {
	Logger *logger.Logger
	Health *health.Service
	// Accounts authenticates bearer tokens.
	//
	// NIL SERVES THE ENTIRE API TO UNAUTHENTICATED CALLERS, including the
	// irreversible POST /admin/purge. It is a development convenience and
	// nothing else, so it must be opted into by name with AllowAnonymous.
	// Without that, APIHandler panics and NewAPIHandler returns an error —
	// mirroring the standalone binary, which refuses to boot in this state.
	Accounts serviceaccount.Authenticator
	// AllowAnonymous opts a deployment into serving the whole API without
	// authentication. It exists so that state cannot be reached by omission.
	//
	// The standalone binary has FLEXITYPE_DEV_INSECURE for the same purpose.
	// Library mode had no equivalent: the fail-closed default was added to
	// internal/config, which only governs the binary, so an embedder who read
	// the release note about authentication becoming fail-closed would
	// reasonably have assumed it applied to them.
	AllowAnonymous bool
	// Metrics, when set, records HTTP SLIs and serves /metrics. With the
	// outbox on, delivery-depth gauges are registered automatically.
	Metrics *metrics.Metrics
	// EnableProvisioning turns on the admin-scoped tenant/service-account
	// API (database-backed only).
	EnableProvisioning bool
	// RateLimiter, when set, throttles API requests per service account
	// (429 + Retry-After). Build one with ratelimit.New.
	RateLimiter *ratelimit.Limiter
	// TenantRateLimiter, when set, caps a tenant's aggregate request rate
	// across all of its service accounts.
	TenantRateLimiter *ratelimit.Limiter
	// AuthRateLimiter, when set, throttles by client address BEFORE
	// authentication. The other two limiters key on a resolved principal, so
	// neither can throttle a FAILED authentication — and each of those costs
	// a database round trip and a hash, uncached, so an unauthenticated
	// caller could exhaust the pool and brute-force tokens unthrottled.
	//
	// Behind a proxy this keys on the proxy, giving a ceiling on aggregate
	// unauthenticated traffic rather than a per-client one. It deliberately
	// does not read X-Forwarded-For: a header is attacker-supplied, so
	// trusting it would let one client spread its attempts across unlimited
	// keys.
	AuthRateLimiter *ratelimit.Limiter
	// DisableConsole omits the admin-console SPA, for an API-only deployment.
	// An unmatched path then returns a JSON 404 like any other API error.
	DisableConsole bool
	// MaxImportBytes caps a CSV import upload; 0 uses the 16 MiB default.
	// GET /features reports the effective value, so a client can chunk a
	// bulk load against the real ceiling instead of guessing.
	MaxImportBytes int64
	// MaxMediaBytes caps a media upload; 0 uses the 32 MiB default.
	MaxMediaBytes int64
}

APIConfig configures the mountable REST API for embedded deployments.

type DeliveryLoops

type DeliveryLoops struct {
	// Relay expands the outbox and dispatches to in-process hooks.
	Relay bool
	// Worker delivers webhook subscriptions.
	Worker bool
	// Pruner enforces event retention.
	Pruner bool
}

DeliveryLoops selects which delivery loops a process runs, so an API tier and a worker tier can be scaled, autoscaled and drained separately from one image.

No leader election is involved: every loop claims work with a lease and FOR UPDATE SKIP LOCKED, so running one on any number of replicas is safe. The switches exist because ten API replicas polling the outbox every two seconds is load that a scaling decision made for request traffic should not create.

func AllDeliveryLoops

func AllDeliveryLoops() DeliveryLoops

AllDeliveryLoops runs everything — the single-process default.

type Option

type Option func(*options)

Option customises an embedded Service.

func WithBackgroundErrorObserver

func WithBackgroundErrorObserver(fn func(err error)) Option

WithBackgroundErrorObserver observes errors from the background schedulers (the change-set publisher and the events-feed pruner), which would otherwise be dropped silently. Use it to log or meter them.

func WithBlobStore

func WithBlobStore(s blob.Store) Option

WithBlobStore backs media attribute values with an object store (local disk, S3-compatible, …). Without it, media uploads return a validation error.

func WithCleanupObserver

func WithCleanupObserver(fn func(err error)) Option

WithCleanupObserver observes swallowed post-erasure cleanup failures — a media-blob GC or search-projection removal that could not be completed after a committed erasure. These are best-effort by design (they must not undo a durable erasure), so use this to log or meter them rather than lose them. Media-blob failures are additionally reported in PurgeReport.MediaBlobsFailed / UnpurgedBlobKeys.

func WithDeadLetterRetention

func WithDeadLetterRetention(d time.Duration) Option

WithDeadLetterRetention bounds how long a DEAD delivery is kept (default 30 days). Only meaningful with WithOutbox.

The envelope prune keeps anything a dead delivery references, which is what makes a dead letter redrivable — but nothing else deleted a dead row, so one decommissioned endpoint pinned its envelopes for ever and the event retention stopped bounding the outbox or the feed at all. This is where that bound lives. It is far longer than the event retention on purpose: a dead letter has to outlive the events it references long enough for an operator to notice it.

func WithDeliveryWorker

func WithDeliveryWorker(opts ...webhook.WorkerOption) Option

WithDeliveryWorker customises the webhook delivery worker (attempt cap, concurrency, HTTP client). Only meaningful with WithOutbox.

func WithDispatchObserver

func WithDispatchObserver(fn func(ctx context.Context, err error)) Option

WithDispatchObserver observes synchronous post-commit event-dispatch failures. In the default (non-outbox) mode the write is already durable when subscribers run, so a subscriber error is reported here instead of failing the request. Use WithOutbox for at-least-once delivery guarantees.

func WithEventRetention

func WithEventRetention(d time.Duration) Option

WithEventRetention sets how long expanded events stay readable in the feed before pruning (default 7 days). Only meaningful with WithOutbox.

func WithFailClosedACL

func WithFailClosedACL() Option

WithFailClosedACL inverts the field-ACL default: a context that carries no uow.Access policy denies every attribute instead of granting admin.

The standalone service always stamps a policy from the authenticated service account, so this option is for embedders. In library mode the host is responsible for stamping the policy on every request, and nothing otherwise enforces that it did — a background job or a new resolver that forgets silently runs with full field access. With this option it fails instead.

Stamp uow.WithAccess on every request path, and uow.WithSystemAccess on host-owned background work that legitimately has no principal. The setting applies to the whole process and cannot be undone; see uow.RequireAccessPolicy.

func WithGraphQLFederation

func WithGraphQLFederation() Option

WithGraphQLFederation exposes the GraphQL endpoint as an Apollo-Federation subgraph: `_service { sdl }`, `_entities(representations:)`, and `@key(fields: "entityId")` on every entity type.

Without it the endpoint is a standalone schema that a federated gateway cannot compose at all. With it, a gateway resolves an entity this service holds attributes for from the entity id another subgraph already owns, which is the natural modelling for an attribute service.

It is off by default: a federated schema carries three fields no standalone client asks for, and `_entities` is a batch read a non-federated deployment has no reason to expose.

func WithHandler

func WithHandler(h events.Handler, opts ...events.RegisterOption) Option

WithHandler registers a dispatcher hook: any events.Handler your infrastructure provides.

func WithHandlerFunc

func WithHandlerFunc(name string, fn func(ctx context.Context, env events.Envelope) error, opts ...events.RegisterOption) Option

WithHandlerFunc registers a plain function hook.

func WithOutbox

func WithOutbox(opts ...outbox.RelayOption) Option

WithOutbox upgrades event delivery to at-least-once: envelopes persist in the same transaction as the change and a relay dispatches them with retries. It also unlocks the standalone-consumer surface — webhook subscriptions and the events feed. Run the delivery machinery with Service.RunOutboxRelay.

func WithPublisher

func WithPublisher(name string, pub events.Publisher, topicFn events.TopicFunc) Option

WithPublisher routes events into your pub/sub broker (NATS, Kafka, SNS, ...). topicFn may be nil to use the event type as the topic.

func WithRollbackObserver

func WithRollbackObserver(fn func(ctx context.Context, err error)) Option

WithRollbackObserver observes rolled-back units of work.

func WithSearchIndex

func WithSearchIndex() Option

WithSearchIndex enables the entity search projection: an internal-projection subscriber keeps one searchable document per entity, unlocking FQL matches(). The index is maintained synchronously in the writing request (read-your-writes) in both delivery modes, so it stays fresh independent of WithOutbox (#211).

func WithTimeZone

func WithTimeZone(loc *time.Location) Option

WithTimeZone sets the calendar day that `today` and `now` resolve against in dependency conditions and dynamic defaults. Default UTC.

It changes which day those name, not how anything is stored: a date value is a calendar date held as midnight UTC either way. Without it, a tenant operating outside UTC had a date-boundary rule that was wrong for part of every day — a condition on "expires before today" flipped at the wrong hour, and a `today` default recorded yesterday for anything created after the UTC midnight.

Per-request override: stamp uow.WithTimeZone on the context, which is how an embedder serves tenants in different zones from one process.

func WithWebhook

func WithWebhook(name string, cfg events.WebhookConfig, opts ...events.RegisterOption) Option

WithWebhook delivers events as signed JSON POSTs to a receiving endpoint.

func WithWebhookAllowPrivate

func WithWebhookAllowPrivate() Option

WithWebhookAllowPrivate lets webhook subscriptions target private, loopback and link-local hosts over http — for on-prem deployments whose consumers live on internal networks. Off by default (SSRF guard).

func WithWebhookTimeout

func WithWebhookTimeout(d time.Duration) Option

WithWebhookTimeout bounds one webhook delivery attempt (default 10s).

It is a duration rather than an *http.Client on purpose: the delivery client is the SSRF guard, and supplying a client would replace that guard without saying so.

func WithoutActivityLog

func WithoutActivityLog() Option

WithoutActivityLog disables the audit log entirely: no pre-commit writes, no read API.

func WithoutSearch

func WithoutSearch() Option

WithoutSearch disables the FQL query surface for this deployment.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service is an embedded flexitype instance.

func New

func New(pool *sqlx.DB, opts ...Option) *Service

New wires an embedded flexitype over your connection pool. The pool is shared, never owned: closing it remains your call.

func NewInMemory

func NewInMemory(opts ...Option) *Service

NewInMemory wires flexitype over the in-memory store: no database, no migrations. Same usecases, same API, same hooks — it powers the browser playground and makes a zero-dependency test double for embedding consumers. Data lives for the process only; WithOutbox is ignored (direct dispatch is already synchronous and in-process).

func (*Service) APIHandler

func (s *Service) APIHandler(cfg APIConfig) http.Handler

APIHandler returns flexitype's versioned REST API as an http.Handler you can mount in your own router.

It PANICS when the configuration would serve the API to unauthenticated callers without an explicit opt-in — that is a composition-time misconfiguration, so it fails at startup rather than per request. Use NewAPIHandler to handle it as an error instead.

func (*Service) AdminInteractor

func (s *Service) AdminInteractor(opts ...admin.Option) *admin.Interactor

AdminInteractor returns the provisioning usecases over this service's pool, or nil for in-memory services.

opts are passed through; APIHandler wires admin.WithAuthCache when the deployment authenticates through a caching authenticator, so a rotation or a revocation takes effect at once rather than at the end of the cache TTL.

func (*Service) BootstrapAdmin

func (s *Service) BootstrapAdmin(ctx context.Context, tenantName, accountName string) (string, error)

BootstrapAdmin seeds the provisioning tables with a tenant and an admin-scoped service account when no accounts exist yet, returning the one-time token so an operator can call the admin API. It is idempotent: once any account exists it returns an empty token and does nothing. This is the only way to get the first credential into a database-backed deployment.

func (*Service) Context

func (s *Service) Context(ctx context.Context) context.Context

Context returns ctx with the service-wide defaults stamped on it: the deployment's time zone, when one is configured.

It exists because those defaults have to travel on the context the CALLER passes to each interactor method. Stamping them inside Interactors derived a context that was then discarded, so FLEXITYPE_TIMEZONE never reached rule evaluation and every `today`/`now` dependency rule and dynamic default resolved in UTC — the read and write paths agreeing only because both were wrong. The API stamps the same defaults in its middleware, which owns the request context.

A caller that already chose a zone keeps it, which is how a host serves tenants in different zones from one process.

func (*Service) Dispatcher

func (s *Service) Dispatcher() *events.Dispatcher

Dispatcher exposes the event dispatcher, for inspection and for registering hooks.

Late registration is safe: the dispatcher copies its handler slice on write under an RWMutex, so a Register concurrent with a Dispatch cannot race. (This comment previously said the opposite, and had done since before the copy-on-write change.)

func (*Service) EnsureWebhookSubscription

func (s *Service) EnsureWebhookSubscription(ctx context.Context, name, url, secret string, eventTypes ...string) error

EnsureWebhookSubscription upserts a webhook subscription by name — the bootstrap path for environment-configured endpoints. Errors unless WithOutbox is enabled.

func (*Service) Factory

func (s *Service) Factory() application.Factory

Factory exposes the underlying usecase factory for advanced wiring.

func (*Service) GraphQLEngine

func (s *Service) GraphQLEngine() *gql.Engine

GraphQLEngine exposes the read-only GraphQL engine, for embedders that build their own API handler (e.g. the WASM playground).

func (*Service) Interactors

func (s *Service) Interactors(ctx context.Context) *application.Interactors

Interactors returns a request-scoped usecase set. Call once per request or unit of work so dataloader caches stay request-local.

PASS THE CONTEXT THROUGH Context FIRST when the deployment sets a time zone. An interactor set carries no context of its own — every method takes one from its caller — so a zone stamped here would reach nothing:

ctx = svc.Context(ctx)
it := svc.Interactors(ctx)
schema, err := it.TypeDefinitions().EffectiveAttributes(ctx, typeID)

func (*Service) Migrate

func (s *Service) Migrate(ctx context.Context) error

Migrate applies flexitype's embedded schema migrations. Safe to call on every startup; concurrent callers serialize on an advisory lock. No-op for in-memory services.

func (*Service) NewAPIHandler

func (s *Service) NewAPIHandler(cfg APIConfig) (http.Handler, error)

NewAPIHandler is APIHandler with the configuration check reported as an error rather than a panic.

func (*Service) NewAccountLookup

func (s *Service) NewAccountLookup(ttl time.Duration) serviceaccount.Authenticator

NewAccountLookup returns a database-backed authenticator over this service's pool, with a short success cache so revocation propagates within ttl. nil for in-memory services.

func (*Service) RecomputeComputed

func (s *Service) RecomputeComputed(ctx context.Context, tenant valueobjects.TenantID) (int, error)

RecomputeComputed re-materializes every entity's computed attributes for a tenant — the recovery counterpart to ReindexSearch. Internal projections are maintained in the originating request's post-commit (issue #211), so a process crash between commit and that post-commit can leave a computed value stale; this rebuilds them all. Returns the number of entities recomputed.

func (*Service) ReindexSearch

func (s *Service) ReindexSearch(ctx context.Context, tenant valueobjects.TenantID) (int, error)

ReindexSearch rebuilds every entity search document for a tenant. Errors unless WithSearchIndex is enabled.

func (*Service) RunChangeSetScheduler

func (s *Service) RunChangeSetScheduler(ctx context.Context, interval time.Duration)

RunChangeSetScheduler publishes approved change-sets whose publish_at has arrived, on the given interval, until ctx ends. Run it as a goroutine next to the server; every replica runs it safely (a published set is skipped by the others). A zero interval defaults to one minute.

func (*Service) RunOutboxRelay

func (s *Service) RunOutboxRelay(ctx context.Context, loops ...DeliveryLoops)

RunOutboxRelay runs the event-delivery machinery until ctx ends: the outbox relay (expansion + in-process dispatch), the webhook delivery worker and the retention pruner. No-op without WithOutbox. Run it as a goroutine next to the server; every replica runs it safely.

func (*Service) SchemaDrift

func (s *Service) SchemaDrift(ctx context.Context) ([]int, error)

SchemaDrift reports migration versions the database has applied that this binary does not carry — the schema is newer than this build.

A rolling deploy makes that state normal for a while: the first new pod migrates while the previous generation keeps serving. flexitype supports it (each release's migrations stay compatible with the previous binary, see docs/upgrades.md), but an operator should be able to see a mixed-version fleet rather than infer it. It returns nothing for an in-memory service and nothing when the schema matches.