Help us shape the future of Go! Take the 2026 Go Developer Survey

dynago

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 30, 2026 License: MIT Imports: 26 Imported by: 0

README

dynago

A schema file for DynamoDB, and the review surface for its design. Declare your entities, exactly how they are read and written, what each read needs, and roughly how much data and traffic to expect. dynago then generates:

  • typed Go access code on guregu/dynamo: one method per declared access pattern and nothing else;
  • a data model document (*.model.md, with Mermaid diagrams) that someone who has never used DynamoDB can review: the domain and its relationships, the rules the code guarantees, each status field's lifecycle, every read and write, and which items share partitions;
  • an analysis of the design: every partition's size, growth and busiest key at peak, the cost of each call, and findings with rule ids (hot keys, keys with too few values, indexes that silently drop items, transaction and size limits), which a schema can accept with a reason and a policy file can enforce;
  • architecture diffs for pull requests (dynago diff), and a check that no code reaches DynamoDB around the schema (dynago vet);
  • infrastructure: Terraform JSON and a create-table definition;
  • a lock file recording each entity's storage shape per version, so schema changes are explicit and migrations can be planned.

It keeps what DynamoDB is good at (predictable single-request reads, horizontal scale) and addresses what makes it hard to work with: the data model is invisible and scattered through code, key formats live in string concatenation, partitions grow and heat up unnoticed, denormalised copies drift out of sync, counts are expensive or wrong, and rules spanning items race.

schema (YAML) ──► Go store + errors          (<table>_dynago.go)
              ├─► model document             (<table>.model.md)
              ├─► Terraform + table JSON     (<table>.tf.json, <table>.table.json)
              ├─► lock / version history     (<table>.dynago.lock)
              ├─► analysis: costs, partitions, findings   (dynago check)
              └─► architecture diff          (dynago diff)

The example, examples/toollibrary, is a fictional neighbourhood tool library that uses most features. Read its schema, the model document generated from it, the generated Go. dynago's end-to-end tests run that generated store against DynamoDB Local.

A taste

Loan:
  doc: A member borrowing a tool.
  fields:
    libraryId: string
    toolId: string
    loanId: string
    memberId: string
    status: { type: enum, values: [active, returned] }
    dueAt: { type: time, required: true }
    returnedAt: time
  key:
    pk: "LIB#{libraryId}#TOOL#{toolId}"   # a tool's loans live together…
    sk: "LOAN#{loanId}"
  volume: { typical: 20, max: 500 }       # …20 per tool, 500 for the busiest
  indexes:
    Overdue:                      # a GSI of active loans by due date…
      pk: "LIB#{libraryId}#DUE"
      sk: "{dueAt}#{loanId}"
      where: { status: active }   # …returned loans drop out
      project: [memberId]
  counters:
    MemberLoans:                  # kept exact by every write, in the same transaction
      pk: "LIB#{libraryId}#MEMBER#{memberId}"
      sk: "LOANS"
      values:
        active: { count: true, where: { status: active }, limit: arg }
  access:
    Overdue: { query: Overdue, range: dueAt, freshness: eventual }   # a GSI will do
    ActiveLoans: { counter: MemberLoans }
  writes:
    Borrow:
      create: true
      set: { status: active }     # whatever the caller passes
      requires:                   # all in the same transaction
        Member: { key: { libraryId: libraryId, memberId: memberId }, when: { status: active } }
        Tool:
          key: { libraryId: libraryId, toolId: toolId }
          when: { status: available }
          set: { status: onLoan }  # borrowing marks the tool on loan, atomically
    Return:
      update: [returnedAt]
      set: { status: returned }
      when: { status: active }
      requires:
        Tool: { key: { libraryId: libraryId, toolId: toolId }, when: { status: onLoan }, set: { status: available } }

st := toollibrary.New(db, toollibrary.TableName("prod-toollibrary"))   // prod-toollibrary-g3

err := st.Loans.Borrow(ctx, loan, toollibrary.LoanBorrowLimits{MemberLoansActive: dynago.Max(member.MaxLoans)})
switch {
case errors.Is(err, toollibrary.ErrMemberLoansActiveLimit): // return something first
case errors.Is(err, toollibrary.ErrLoanBorrowRequiresTool): // the tool is out
}

overdue, next, err := st.Loans.Overdue(ctx, toollibrary.LoanOverdueQuery{LibraryID: lib, To: &now}, dynago.Page{})

Borrow writes the loan, marks the tool on loan (and updates the library's tool counts), adds to the member's active count (conditioned on their limit) and checks that the member is active, all in one transaction, without reading anything first. There is no moment at which a loan exists for an available tool, and no crash that can leave one. Return reads the loan first (it leaves the member's list of current loans), then reverses the tool and the count in one transaction. The tests race ten members for one tool and assert that exactly one gets it, and check that a refused borrow leaves the table byte-for-byte unchanged.

Install and use

go get -tool github.com/nicklanng/dynago/cmd/dynago   # pins the generator in go.mod (Go 1.25+)

go tool dynago check    things.dynago.yaml            # validate + analysis: costs, partitions, findings
go tool dynago generate things.dynago.yaml            # write the Go, docs, Terraform, table JSON and lock
go tool dynago generate -check things.dynago.yaml     # CI: fail if any generated file is stale
go tool dynago diff -base origin/main things.dynago.yaml   # a pull request's architectural changes
go tool dynago vet ./...                              # CI: no DynamoDB calls around the schema

Commit the schema and everything generated, including the lock file. The model document is the review surface for the design, and dynago diff for each change to it (reviewing designs and changes). The getting started guide walks through it end to end.

What it gives you

  • Only declared reads and writes exist. Each read is one request. A query nobody declared has no method, so every new way of reading data shows up in review as a schema change. dynago vet finds code that calls DynamoDB around the store; a deliberate exception is marked with its reason, and a lasting one is declared in the schema (a scan with a reason).
  • The design is analysed. From a few numbers per entity (a total, or typical and max per parent), dynago estimates each partition's item count, size, growth and busiest key at peak, and reports the risks: hot partitions and counters, keys with too few values, indexes that drop items without a word, lookups that assume uniqueness nothing enforces, copies whose fan-out one transaction can't hold. Findings have rule ids: accept one on the object it's about with a reason, or set a dynago.policy.yaml for the repository.
  • Requirements, not just mechanisms. A read states the freshness it needs, and dynago makes its index a GSI or a transactional copy accordingly, showing what the other would cost. A copied field says whether it's a snapshot_of its source or a copy_of that must stay equal.
  • Changes are reviewed as architecture. dynago diff turns a schema change into what it means: a new index and the writes it now adds to, a read that became strongly consistent, a partition that grows, a new risk, and whether existing items need a new table generation.
  • Derived items stay consistent. Copies (a transactional index, readable immediately), uniqueness claims (one per value, or per element of a set), and counters with upper (limit) and lower (min) bounds change atomically with the item, and every write path computes them the same way. GSIs are maintained by DynamoDB, asynchronously: usually well under a second behind.
  • Rules and changes across entities (requires) happen in the write's transaction: check another item, change it (set), delete it (consume), or check a counter ("no active loans"). Nothing can change in between, and nothing is left half done.
  • Writes cost what they must. An update that feeds nothing derived is one UpdateItem. One whose counter change is known from its arguments (set: { status: retired }, when: { status: available }) runs without reading the item, and so do changes to other entities under the same rule. Only the rest read first.
  • Optional updates (patch:) change only the fields given, and read first only when a given field feeds something derived.
  • Required fields (required: true) can't be written empty.
  • Every row is timestamped: _created and _updated on every item dynago writes, set on every write path, and read as e.Timestamps().
  • Lost updates prevented with document versions: Version(), dynago.From, dynago.IfVersion, and versioned: required writes.
  • Safe to run in production:
    • read-modify-writes are guarded by a revision that starts at a random value;
    • a transaction or create that the SDK retries after it succeeded is not applied twice or reported as failed (a single-item update or delete retried that way may report a failed condition for a change that did go through);
    • during a rolling deploy or after a rollback, older code keeps the fields a newer version stored when it rewrites an item;
    • TTL-expired items are treated as gone before DynamoDB deletes them;
    • key values can't collide through separators;
    • cursors can be signed.
  • Schema changes are explicit, and migrations are generated. Entities have versions, and a lock file records every shape. A change existing items don't fit (a new index, claim or counter, a new key) moves the table to a new generation. A generated job copies the old table into the new one while the old version serves, catches up, and makes a short final pass with writes stopped. The old table stays for rollback. See Migrations.

Documentation

Testing your store

dynagotest connects to DynamoDB Local and gives each test its own table:

db := dynagotest.DB(t)   // skips unless DYNAGO_TEST_ENDPOINT is set
st := toollibrary.New(db, dynagotest.Table(t, db, toollibrary.TableSpec))

dynagotest.CountingDB also counts the requests a call makes, reads and writes. In this repository, make test starts DynamoDB Local in Docker and runs everything (with DYNAGO_REQUIRE_DB=1, so nothing skips silently); make test-unit needs no Docker.

Status and roadmap

Working now: everything above, exercised by end-to-end tests of the example against DynamoDB Local.

Next:

  1. LSIs (strategy: lsi). Now that a new table generation can add or drop them, they're no longer a decision you can't undo. The partition analysis already estimates the item collection sizes their 10 GB limit needs.
  2. Importing from another table, such as a legacy single table, through the same job engine.
  3. Runtime checks. Compare consumed capacity per access pattern with the estimates, so the declared volumes and rates stay true.

Known limits:

  • TTL expiry bypasses the generated code. An expired item does not decrement its counters or release its claims and copies. Reads and requires treat it as gone, and dynago check warns about each case. Design expiring items to own nothing else, as the example's holds do.
  • Background writes change an item's version too, so a form left open on an item that jobs update often will see conflicts.
  • Migrations read the whole table on every pass. For very large, old tables, a table copy per structural change may stop being practical.
  • Not supported yet: LSIs (see the roadmap), custom Go field types, batch writes, multi-valued index keys.

License

MIT. See LICENSE.

Documentation

Overview

Package dynago is the runtime for code generated by the dynago tool.

dynago turns a schema file describing a DynamoDB table (its entities, keys, indexes, uniqueness claims, counters, and every read and write) into a typed Go store, a data model document, Terraform and a cost report. The generated store calls this package; application code uses it for the few things that cross stores:

  • sentinel errors such as ErrNotFound, ErrTaken, ErrLimit and ErrVersionMismatch, which every generated error wraps;
  • Page, to page through queries;
  • Limit and Max, for counter limits supplied by the caller;
  • From and IfVersion, for optimistic concurrency.

The remaining exported identifiers (Op, Diff, Run, QuerySpec and so on) are building blocks for generated code. They are stable, but not meant to be called by hand.

Documentation: https://github.com/nicklanng/dynago/tree/main/docs

Example

A store is created from a guregu/dynamo DB and a table name; its fields hold one store per entity. 

package main

import (
	"context"
	"errors"
	"fmt"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/guregu/dynamo/v2"

	"github.com/nicklanng/dynago/examples/toollibrary"
)

// These examples use the store generated from examples/toollibrary/toollibrary.dynago.yaml.
var ctx = context.Background()

func main() {
	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		panic(err)
	}
	st := toollibrary.New(dynamo.New(cfg), "toollibrary-prod")

	tool, err := st.Tools.Get(ctx, toollibrary.ToolKey{LibraryID: "lib_1", ToolID: "t_42"})
	switch {
	case errors.Is(err, toollibrary.ErrToolNotFound):
		fmt.Println("no such tool")
	case err != nil:
		panic(err)
	default:
		fmt.Println(tool.Name, tool.Status)
	}
}

Index

Examples

Constants

View Source 
const (
	AttrPK   = "PK"
	AttrSK   = "SK"
	AttrType = "_t"
	AttrVer  = "_v"
	AttrRev  = "_rev"
	// AttrCreated and AttrUpdated hold when dynago first wrote an item and when it last changed
	// it, in TimeLayout. Items written before dynago kept them lack them until rewritten.
	AttrCreated = "_created"
	AttrUpdated = "_updated"
)

Attribute names written on every item.

View Source 
const MigrateUsage = `` /* 475-byte string literal not displayed */

MigrateUsage describes the migration command's arguments, for generated commands.

View Source 
const RangeEnd = "\U0010FFFF"

RangeEnd is appended to a sort key prefix to form an inclusive upper bound that sorts after every key starting with that prefix.

View Source 
const TimeLayout = "2006-01-02T15:04:05.000000000Z"

TimeLayout is the fixed-width UTC layout used for times in keys, so they sort chronologically.

Variables

View Source 
var (
	// ErrNotFound means the item does not exist.
	ErrNotFound = errors.New("dynago: not found")
	// ErrExists means a create found an item already at that key.
	ErrExists = errors.New("dynago: already exists")
	// ErrPrecondition means a write's declared condition did not hold: an update's `when`, or
	// something it `requires` of another item.
	ErrPrecondition = errors.New("dynago: precondition failed")
	// ErrTaken means a unique value is already claimed by another item.
	ErrTaken = errors.New("dynago: unique value already taken")
	// ErrLimit means a write would take a limited counter past its limit.
	ErrLimit = errors.New("dynago: counter limit reached")
	// ErrConflict means the item kept changing under a read-modify-write until retries ran out.
	ErrConflict = errors.New("dynago: concurrent modification")
	// ErrInvalidCursor means a page cursor was malformed or belongs to a different query.
	ErrInvalidCursor = errors.New("dynago: invalid cursor")
	// ErrInvalidKey means a key field was empty, or contained a character its key uses as a
	// separator.
	ErrInvalidKey = errors.New("dynago: invalid key")
	// ErrFieldRequired means a write left a field declared `required: true` at its zero value.
	ErrFieldRequired = errors.New("dynago: required field is empty")
	// ErrLimitRequired means a write that takes a caller-supplied limit was not given one.
	ErrLimitRequired = errors.New("dynago: limit required")
	// ErrSameItemTwice means a write would touch one item twice in a transaction, which DynamoDB
	// rejects. It is a schema shape dynago should have refused when generating: please report it.
	ErrSameItemTwice = errors.New("dynago: a transaction would touch the same item twice")
	// ErrTooManyItems means a write would touch more than DynamoDB's 100-item transaction limit.
	ErrTooManyItems = errors.New("dynago: write touches more than 100 items")

	// ErrNeedsRead is returned internally when a write tried without reading the item found that
	// the state it assumed did not hold. Generated writes then read the item and try again; callers
	// never see it.
	ErrNeedsRead = errors.New("dynago: write must read the item first")

	// ErrStale is returned internally when an item changed between a read and the conditional
	// write that followed it. Generated writes retry on it; callers see ErrConflict if retries run out.
	ErrStale = errors.New("dynago: item changed since it was read")
	// ErrContention is returned when DynamoDB cancels a transaction because another transaction
	// touched the same item. Generated writes retry on it.
	ErrContention = errors.New("dynago: transaction conflict")
)

Sentinel errors. Generated code wraps these in entity-specific errors, so both errors.Is(err, dynago.ErrNotFound) and errors.Is(err, mytable.ErrEventNotFound) work.

View Source 
var (
	// ErrVersionMismatch means the item changed since the version the caller supplied: someone
	// else wrote it in between. Writes never retry it; the caller decides (reload, merge, report).
	ErrVersionMismatch = errors.New("dynago: item changed since the version supplied")
	// ErrVersionRequired means the write is declared `versioned: required` and no version was
	// supplied, or the entity passed with From was not read from the store.
	ErrVersionRequired = errors.New("dynago: this write requires the version that was read")
)
View Source 
var DefaultRetries = RetryPolicy{Attempts: 8, BaseDelay: 5 * time.Millisecond, MaxDelay: 250 * time.Millisecond}

DefaultRetries is the policy generated writes use unless SetRetries changes it: 8 attempts, waiting 5 ms, 10 ms, ... up to 250 ms between them, about half a second in total. That rides out bursts of contention on a busy counter.

View Source 
var ErrLeaseLost = errors.New("dynago: another migration job holds the lease")

ErrLeaseLost means another migration job took over: this one stops without writing more.

View Source 
var ErrMigrationConflict = errors.New("dynago: item can't be copied")

ErrMigrationConflict is wrapped by errors a migration reports as conflicts: items it can't copy until a person fixes the data in the old table (two old items converting to one new key, a value a conversion rejects). Claims taken, invalid keys and empty required fields are conflicts too. Other errors stop the job.

View Source 
var ErrUnchanged = errors.New("dynago: already copied at this revision")

ErrUnchanged is returned by a migration's Copy when the new table already holds the item at the source's revision, or when the item isn't copied at all (it has expired).

Functions

func CheckAbsent 

func CheckAbsent(t dynamo.Table, key Key, ttlAttr string) *dynamo.ConditionCheck

CheckAbsent builds a condition check that an item is absent or expired. A write that read an optional item and found none uses it to fail if one appeared before the write commits.

func CheckCounter 

func CheckCounter(t dynamo.Table, key Key, values []Cond) *dynamo.ConditionCheck

CheckCounter builds a condition check on a counter item's values. A missing value (or item) counts as zero.

func CheckKeyPart 

func CheckKeyPart(field, value, separators string) error

CheckKeyPart rejects a value that contains a character the key template uses to separate its parts: "a#b" + "c" and "a" + "b#c" would otherwise render the same key. The value is checked as given and as {field|lower} renders it, since lower-casing (and Unicode normalisation) can produce a separator: "AX" before a literal "x", or the Kelvin sign becoming "k".

func CheckRequirement 

func CheckRequirement(t dynamo.Table, r Requirement) *dynamo.ConditionCheck

CheckRequirement builds a condition check that another item meets a requirement.

func CondUpdate 

func CondUpdate(u *dynamo.Update, c Cond)

CondUpdate adds an equality precondition to an update.

func ConsumeRequirement 

func ConsumeRequirement(t dynamo.Table, r Requirement) *dynamo.Delete

ConsumeRequirement builds the deletion of another item that meets a requirement.

func DeleteIfExists 

func DeleteIfExists(ctx context.Context, t dynamo.Table, key Key, g Guard) error

DeleteIfExists deletes an item that has no derived items, returning the Guard's errors if it is absent, expired or at another revision.

func Distinct 

func Distinct(s []string) []string

Distinct returns s without repeated values, keeping the first of each: a list converted to a set, which can't hold duplicates.

func EnsureTable 

func EnsureTable(ctx context.Context, db *dynamo.DB, name string, spec TableSpec) error

EnsureTable creates the table if it does not exist, waits for it to become active and enables TTL. It is meant for local development and tests; production tables belong in infrastructure code (dynago generates Terraform for them).

func Expired 

func Expired(ttl, now int64) bool

Expired reports whether a TTL attribute value (epoch seconds, 0 for none) has passed. DynamoDB deletes expired items lazily, often hours later, so reads must check.

func ExpiredItem 

func ExpiredItem(raw dynamo.Item, ttlAttr string) bool

ExpiredItem reports whether a stored item's TTL attribute has passed.

func FmtBool 

func FmtBool(b bool) string

FmtBool renders a bool for a key.

func FmtInt 

func FmtInt(i int64) string

FmtInt renders an integer for a key so that lexical order matches numeric order.

func FmtStamp 

func FmtStamp(t time.Time) string

FmtStamp renders a timestamp for storage; the zero time is "".

func FmtString 

func FmtString[S ~string](s S) string

FmtString renders a string for a key.

func FmtTime 

func FmtTime(t time.Time) string

FmtTime renders a time for a key.

func FormatVersion 

func FormatVersion(rev int64) string

FormatVersion renders a revision as an opaque version string, suitable for an ETag.

func GetMany 

func GetMany(ctx context.Context, t dynamo.Table, keys []Key, consistent bool, out any) error

GetMany reads items by key with one BatchGetItem (chunked by the library at 100 keys) and appends the ones that exist to out, which must be a pointer to a slice.

func GetOne 

func GetOne(ctx context.Context, t dynamo.Table, key Key, consistent bool, out any) (found bool, err error)

GetOne reads one item into out. It reports found=false rather than an error when the item does not exist.

func GuardUpdate 

func GuardUpdate(u *dynamo.Update, g Guard, now int64)

GuardUpdate adds a Guard's conditions to an update.

func ItemRev 

func ItemRev(raw dynamo.Item) int64

ItemRev returns the revision of a raw item.

func ItemType 

func ItemType(raw dynamo.Item) string

ItemType returns the entity or derived-item type (_t) of a stored item.

func KeepUnknown 

func KeepUnknown(raw dynamo.Item, known map[string]bool, item any) (dynamo.Item, error)

KeepUnknown returns item, marshalled, with every attribute of raw (the stored item it replaces) that known doesn't name. Code at an older compatible version rewriting an item that newer code wrote must not drop the fields it doesn't know: within a table generation, versions only add or remove plain fields, so keeping what it doesn't know is always safe.

func Lower 

func Lower(s string) string

Lower is the `lower` key template transform: {name|lower} sorts and matches case-insensitively. It normalises to NFC first, so an "é" typed as one code point and as "e" plus a combining accent render the same key. Keys still sort by bytes, not by any language's alphabet: "émile" sorts after "zoe".

func MigratedFrom 

func MigratedFrom(raw dynamo.Item, src Key, srcRev int64) (same, other bool)

MigratedFrom inspects an item of the new table: same reports whether it was copied from src at srcRev; other reports whether it was copied from a different source item (two old items converting to one new key).

func MigrationItem 

func MigrationItem(item any, src Key, srcRev int64) (dynamo.Item, error)

MigrationItem returns item, marshalled, with the source item's key and revision recorded on it.

func NeedsRead 

func NeedsRead(err error) bool

NeedsRead reports whether a write attempted without reading the item found that its assumptions did not hold, so it must be retried by reading first.

func NewRev 

func NewRev() int64

NewRev returns the starting revision for a new item. It is random rather than 1 so that a read-modify-write guarded by a revision read before a delete and re-create cannot mistake the new item for the one it read. It stays below 2^52 so it survives a trip through a float64 (JSON, JavaScript) exactly.

func NewStamp 

func NewStamp() string

NewStamp returns the current time as dynago stores it in _created and _updated.

func Now 

func Now() int64

Now returns the current time in epoch seconds, for TTL checks.

func ParseMigrate 

func ParseMigrate(args []string, base string) (command, table string, workers, passes int, rate float64, err error)

ParseMigrate reads the migration command's arguments.

func ParseStamp 

func ParseStamp(s string) time.Time

ParseStamp reads a stored timestamp; "" (or anything unreadable) is the zero time.

func ParseVersion 

func ParseVersion(s string) (int64, error)

ParseVersion parses a version string made by FormatVersion.

func Path 

func Path(attr string) string

Path quotes an attribute name for guregu's update builders unless it is a plain identifier. Names such as "_rev" are not valid bare identifiers in DynamoDB expressions; quoting makes guregu substitute a placeholder for them.

func Ptr 

func Ptr[T any](v T) *T

Ptr returns a pointer to v, for optional (patch) update fields: dynago.Ptr("new title").

Example

Ptr makes the pointers that optional (patch) fields take: nil leaves a field unchanged. 

package main

import (
	"context"

	"github.com/nicklanng/dynago"
	"github.com/nicklanng/dynago/examples/toollibrary"
)

// These examples use the store generated from examples/toollibrary/toollibrary.dynago.yaml.
var (
	ctx = context.Background()
	st  *toollibrary.Store
)

func main() {
	key := toollibrary.MemberKey{LibraryID: "lib_1", MemberID: "m_7"}
	// Only the phone number changes; the name is left as it is.
	_ = st.Members.UpdateProfile(ctx, key, toollibrary.MemberUpdateProfile{Phone: dynago.Ptr("555-0100")})
}

func Query 

func Query(ctx context.Context, t dynamo.Table, spec QuerySpec, page Page, out any) (string, error)

Query runs exactly one Query request for one page and appends the items to out, which must be a pointer to a slice. It returns the cursor for the next page, or "" when there are no more.

func ReadIfNeeded 

func ReadIfNeeded(fn func(read bool) error) error

ReadIfNeeded runs a write that also changes other items. It first builds the changes assuming those items are in the state the schema requires, without reading them; if that assumption fails (ErrNeedsRead), it runs again reading them, which reports precisely what was wrong.

func RequireLimit 

func RequireLimit(name string, l Limit) error

RequireLimit returns ErrLimitRequired if l was not given. Generated writes call it for every limit they take.

func Retry 

func Retry(ctx context.Context, fn func() error) error

Retry runs a write, retrying on ErrStale and ErrContention under the retry policy (see SetRetries). After the last attempt it returns ErrConflict.

func Run 

func Run(ctx context.Context, db *dynamo.DB, ops []Op) error

Run applies ops atomically: a single op runs as a plain write (half the cost of a transaction), several run as one TransactWriteItems. A failed condition is reported as the error registered for the op that failed.

func Scan 

func Scan(ctx context.Context, t dynamo.Table, spec ScanSpec, page Page, out any) (string, error)

Scan runs exactly one Scan request for one page and appends the entity's items to out, which must be a pointer to a slice. A page evaluates up to its size of the table's items, of every kind, and keeps the entity's: it can hold few items, or none, and still have a next cursor. It returns the cursor for the next page, or "" when the whole table has been read.

func SetFields 

func SetFields(u *dynamo.Update, sets []Set)

SetFields applies attribute changes to an update, bumps the item's revision and stamps it as updated now.

func SetRetries 

func SetRetries(p RetryPolicy)

SetRetries sets the policy generated writes use. It is safe to call while writes run.

func ShardPK 

func ShardPK(pk string, owner Key, shards int) string

ShardPK appends the shard an owner's contributions land on to a counter partition key.

func ShardPKs 

func ShardPKs(pk string, shards int) []string

ShardPKs returns every shard's partition key, for reading a sharded counter.

func SignCursors 

func SignCursors(key []byte, previous ...[]byte)

SignCursors makes page cursors carry an HMAC-SHA256 under key, so a client cannot forge or edit one. Every instance must use the same key. Cursors signed with any of previous are still accepted, so the key can be rotated: deploy the new key with the old one as previous, then drop the old one once its cursors have expired from clients. SignCursors(nil) turns signing off. It is safe to call while queries run.

Without signing, cursors are only checked to belong to the query, partition and range bounds that produced them.

Example

SignCursors makes page cursors tamper-proof. Call it once at startup with a secret shared by every instance. 

package main

import (
	"github.com/nicklanng/dynago"
)

func main() {
	dynago.SignCursors([]byte("a secret from your secret store"))
}

func StaleAs 

func StaleAs(versioned bool, err error) error

StaleAs reports a write that lost a race with another writer as ErrVersionMismatch when the caller supplied a version (the caller must decide what to do), and leaves it as ErrStale (which Retry retries) otherwise.

func UnixTTL 

func UnixTTL(t time.Time) int64

UnixTTL converts a time to the epoch seconds DynamoDB's TTL expects; zero stays zero.

func UpdateFields 

func UpdateFields(ctx context.Context, t dynamo.Table, key Key, sets []Set, when []Cond, g Guard, precondition error) (int64, error)

UpdateFields changes attributes of an existing item in one UpdateItem, bumping its revision, and returns the new revision. It is used for updates that feed no counters, claims, copies or index keys, so no read is needed. Besides the Guard's errors it returns precondition if a Cond fails.

Types

type Change 

type Change struct {
	Owner         Key
	Before, After []Derived
}

Change is one item's derived items before and after a write. A write that also changes other entities (through requires) has one Change per entity it changes.

type Claim 

type Claim = claimItem

Claim is the stored form of a uniqueness claim.

type Cond 

type Cond struct {
	Attr  string
	Value any
	Zero  bool
}

Cond is an equality precondition. Zero marks the field's zero value, which may be stored as an absent attribute.

type Derived 

type Derived struct {
	Kind DerivedKind
	Key  Key
	// Type is the _t value stored on the derived item.
	Type string

	// Counter contribution.
	Attr     string
	Amount   int64
	Limit    Limit
	LimitErr error
	// LimitRequired marks a value whose limit the caller must supply: growing it without one fails.
	LimitRequired bool
	// Min is a lower bound on the counter value; MinErr is returned when a write would cross it.
	Min    Limit
	MinErr error

	// Claim.
	TakenErr error

	// Copy: the full item to store, including its PK, SK, _t and _v attributes. Created is the
	// owner's creation time (TimeLayout), stamped on the copy; "" for a new owner, stamped now.
	Item    any
	Created string
}

Derived describes an item that exists because of an entity's state. Generated code computes the derived items of an entity before and after a write; Diff turns the difference into writes.

func Unbounded 

func Unbounded(ds []Derived) []Derived

Unbounded returns derived items without their limits and lower bounds. The migration job uses it: it copies what the old table holds, whose writes already enforced its rules, so a count briefly out of range while items are copied in any order is not an error.

type DerivedKind 

type DerivedKind int

DerivedKind is the kind of item maintained alongside an entity item. 

const (
	// KindCounter is one entity's contribution to one attribute of a counter item.
	KindCounter DerivedKind = iota + 1
	// KindClaim is a uniqueness claim owned by the entity item.
	KindClaim
	// KindCopy is a copy item holding a projection of the entity, used as a transactional index.
	KindCopy
)

Kinds of derived items.

type GSISpec 

type GSISpec struct {
	Name   string `json:"name"`
	PKAttr string `json:"pk"`
	// SKAttr is "" for an index without a sort key.
	SKAttr string `json:"sk,omitempty"`
	// Projection is "ALL" or "INCLUDE".
	Projection  string   `json:"projection"`
	NonKeyAttrs []string `json:"non_key_attributes,omitempty"`
}

GSISpec is one global secondary index.

type Guard 

type Guard struct {
	// ExpectRev, if non-zero, is the revision the caller read (a document version).
	ExpectRev int64
	// TTLAttr, if set, makes an expired item count as absent.
	TTLAttr string
	// NotFound is returned if the item is absent (or expired).
	NotFound error
}

Guard is the conditions every write to an existing item carries.

type Key 

type Key struct {
	PK string
	SK string
}

Key is a table primary key.

func ItemKey 

func ItemKey(raw dynamo.Item) Key

ItemKey returns a stored item's key.

func SourceOf 

func SourceOf(raw dynamo.Item) (Key, int64, error)

SourceOf returns the key and revision of an item of the old table.

type Limit 

type Limit struct {
	Max       int64
	Set       bool
	Unbounded bool
}

Limit is a caller-supplied cap for a counter value declared with `limit: arg`. Build it with Max or Unlimited: the zero Limit is "not given", which generated writes reject with ErrLimitRequired, so a forgotten limit can never silently switch enforcement off.

func Max 

func Max(n int64) Limit

Max returns a Limit of n.

Example

Max caps a counter value declared with `limit: arg`, here how many tools a member may have out. The counter update is conditioned on the limit, so concurrent borrows can never exceed it. A limit must always be given: pass Unlimited to switch it off on purpose. 

package main

import (
	"context"
	"errors"
	"fmt"
	"time"

	"github.com/nicklanng/dynago"
	"github.com/nicklanng/dynago/examples/toollibrary"
)

// These examples use the store generated from examples/toollibrary/toollibrary.dynago.yaml.
var (
	ctx = context.Background()
	st  *toollibrary.Store
)

func main() {
	member, err := st.Members.Get(ctx, toollibrary.MemberKey{LibraryID: "lib_1", MemberID: "m_7"})
	if err != nil {
		panic(err)
	}
	err = st.Loans.Borrow(ctx, &toollibrary.Loan{
		LibraryID: "lib_1", ToolID: "t_42", LoanID: "ln_9", MemberID: "m_7",
		BorrowedAt: time.Now(), DueAt: time.Now().Add(7 * 24 * time.Hour),
	}, toollibrary.LoanBorrowLimits{MemberLoansActive: dynago.Max(member.MaxLoans)})
	switch {
	case errors.Is(err, toollibrary.ErrMemberLoansActiveLimit):
		fmt.Println("return something first")
	case errors.Is(err, toollibrary.ErrLoanBorrowRequiresTool):
		fmt.Println("that tool is out")
	case errors.Is(err, toollibrary.ErrLoanBorrowRequiresHold):
		fmt.Println("that tool is reserved for someone else")
	}
}

func Unlimited 

func Unlimited() Limit

Unlimited returns a Limit that deliberately enforces nothing.

type Migration 

type Migration struct {
	DB       *dynamo.DB
	From, To string
	// Types lists the entity types (their _t values) to copy. Other items of the old table
	// (claims, copies, counters, and entities this generation dropped) are not copied: the new
	// table's derived items are rebuilt from the entities.
	Types map[string]bool
	// Copy writes the entity converted from one item of the old table into the new one, with its
	// derived items and fence (an op the write must include in its transaction). It returns
	// ErrUnchanged if the new table already has that revision, or the item isn't copied.
	Copy func(ctx context.Context, raw dynamo.Item, fence Op) error
	// KeyOf returns the key an item of the old table converts to in the new one, or ok false if
	// it isn't copied (expired, or not an entity of this generation).
	KeyOf func(raw dynamo.Item) (key Key, ok bool, err error)
	// Remove deletes an entity of the new table, with its derived items and fence, because the
	// item it was copied from is gone or now converts to another key. It returns ErrUnchanged if
	// the entity is gone already, or has been rewritten since raw was read (a fresher copy).
	Remove func(ctx context.Context, raw dynamo.Item, fence Op) error
	// Check reports why the migration can't run at all (a conversion function not provided), or nil.
	Check func() error
	// TTLAttr is the TTL attribute: expired items of the old table are not copied.
	TTLAttr string

	Workers   int     // segments scanned at once (and segments per new pass); default 8
	Rate      float64 // items per second across workers; 0 for no limit
	MaxPasses int     // catch-up passes after the first, for copy; default 3
	Out       io.Writer
	// LeaseFor is how long a job holds the lease without renewing it; default 2 minutes. The
	// lease is renewed every third of it.
	LeaseFor time.Duration
}

Migration copies one table generation into the next.

func (*Migration) Run 

func (m *Migration) Run(ctx context.Context, command string) error

Run runs a migration command: "copy" (bulk copy and catch-up passes, while the old generation serves), "finish" (a final pass, with writes to the old generation stopped) or "status".

type Op 

type Op struct {
	// contains filtered or unexported fields
}

Op is one item write, with the error to report if its condition is what failed.

func CheckOp 

func CheckOp(key Key, c *dynamo.ConditionCheck, condErr error) Op

CheckOp wraps a condition check on the item at key. condErr is returned when the check fails.

func CreateOp 

func CreateOp(key Key, p *dynamo.Put, condErr error, rev int64) Op

CreateOp wraps the put that creates the item at key with revision rev. condErr is returned when the item already exists, unless it is this very write repeated by an SDK retry.

func DeleteOp 

func DeleteOp(key Key, d *dynamo.Delete, condErr error) Op

DeleteOp wraps a delete of the item at key. condErr is returned when its condition fails.

func Diff 

func Diff(t dynamo.Table, owner Key, before, after []Derived) ([]Op, error)

Diff returns the writes that move the derived items of owner from before to after. See DiffAll.

func DiffAll 

func DiffAll(t dynamo.Table, changes []Change) ([]Op, error)

DiffAll returns the writes that apply every change of one transaction:

  • counters get one ADD per changed item, conditioned on their limits when they grow. Changes to the same counter item from different owners are merged, since a transaction may touch an item only once (two accounts of one tenant moving between states in one write);
  • claims are created (conditioned on being free or already the owner's) or released;
  • copies are put when new or changed and deleted when gone.

func PutOp 

func PutOp(key Key, p *dynamo.Put, condErr error) Op

PutOp wraps a put of the item at key. condErr is returned when the put's condition fails.

func UpdateOp 

func UpdateOp(key Key, u *dynamo.Update, condErr error) Op

UpdateOp wraps an update of the item at key. condErr is returned when its condition fails.

type OpError 

type OpError struct {
	Key Key
	Err error
}

OpError is a write's failed condition, carrying the key of the item whose condition failed. It unwraps to the error registered for that item, so errors.Is works as with the bare error; errors.As gives the key, which the migration job uses to find who holds a claim.

func (*OpError) Error 

func (e *OpError) Error() string

func (*OpError) Unwrap 

func (e *OpError) Unwrap() error

type Page 

type Page struct {
	// Size is the maximum number of items to evaluate. Zero means the access pattern's default;
	// sizes above its maximum are clamped.
	Size int
	// Cursor is the Next value returned by the previous page, or "" for the first page.
	Cursor string
}

Page selects one page of a query. The zero Page asks for the first page at the default size.

Example

Page selects one page of a query. Every query method makes exactly one request per call and returns the cursor for the next page, "" at the end. 

package main

import (
	"context"
	"fmt"

	"github.com/nicklanng/dynago"
	"github.com/nicklanng/dynago/examples/toollibrary"
)

// These examples use the store generated from examples/toollibrary/toollibrary.dynago.yaml.
var (
	ctx = context.Background()
	st  *toollibrary.Store
)

func main() {
	q := toollibrary.ToolCatalogueQuery{LibraryID: "lib_1", Category: toollibrary.ToolCategoryPower}
	page := dynago.Page{Size: 25}
	for {
		tools, next, err := st.Tools.Catalogue(ctx, q, page)
		if err != nil {
			panic(err)
		}
		for _, t := range tools {
			fmt.Println(t.Name, t.Status)
		}
		if next == "" {
			break
		}
		page.Cursor = next
	}
}

type QuerySpec 

type QuerySpec struct {
	// Scope identifies the access pattern and partition; a cursor is only valid for the scope
	// that produced it.
	Scope string
	// Index is the GSI name, or "" for the base table.
	Index  string
	PKAttr string
	PK     string
	SKAttr string
	// Prefix restricts the query to sort keys starting with it (the item family's prefix).
	Prefix string
	// From and To, when set, bound the sort key inclusively. They already include Prefix.
	From, To   *string
	Desc       bool
	Consistent bool
	PageSize   int
	MaxPage    int
	// TTLAttr, if set, leaves out items whose TTL has passed but that DynamoDB has not yet deleted.
	// Such a page may hold fewer items than its size and still have a next cursor.
	TTLAttr string
	// Project, if set, reads only these attributes. It saves bandwidth and keeps other attributes
	// from callers; DynamoDB still charges for the whole item.
	Project []string
}

QuerySpec describes one generated query.

type Requirement 

type Requirement struct {
	Key Key
	// When lists values the item must have.
	When []Cond
	// TTLAttr, if set, makes an expired item count as absent.
	TTLAttr string
	// Optional lets an absent (or expired) item pass: When applies only to an item that is there.
	Optional bool
}

Requirement is the state a write requires of another item, checked in the write's transaction.

type RetryPolicy 

type RetryPolicy struct {
	Attempts  int
	BaseDelay time.Duration
	// MaxDelay caps each delay; zero means DefaultRetries.MaxDelay.
	MaxDelay time.Duration
}

RetryPolicy controls how generated writes retry when the item changed under them (ErrStale) or another transaction held one of their items (ErrContention). Delays grow exponentially from BaseDelay up to MaxDelay, with jitter: each wait is between half and one and a half times the delay.

type ScanSpec 

type ScanSpec struct {
	// Scope identifies the access pattern; a cursor is only valid for the scope that produced it.
	Scope string
	// Type is the entity's type attribute (_t): the scan returns only its items.
	Type       string
	Consistent bool
	PageSize   int
	MaxPage    int
	// TTLAttr, if set, leaves out items whose TTL has passed.
	TTLAttr string
}

ScanSpec describes one generated scan: a declared read of every item of one entity.

type Set 

type Set struct {
	Attr   string
	Value  any
	Remove bool
	// StringSet stores Value as a DynamoDB string set rather than a list.
	StringSet bool
}

Set is one attribute change in an update. Zero values are removed rather than stored, which matches how items are written (every non-key attribute is omitted when empty).

type TableSpec 

type TableSpec struct {
	TTLAttr string    `json:"ttl_attribute,omitempty"`
	GSIs    []GSISpec `json:"gsis,omitempty"`
}

TableSpec is the physical shape of a table: its GSIs and TTL attribute. The base key is always PK (hash) and SK (range), both strings. Billing is on-demand.

func (TableSpec) CreateTableInput 

func (s TableSpec) CreateTableInput(name string) *dynamodb.CreateTableInput

CreateTableInput builds the CreateTable request for a table named name.

type Timestamps 

type Timestamps struct {
	Created, Updated time.Time
}

Timestamps say when an item was first written and last changed. They come from the clock of the server that wrote it, so across servers they are ordered only as well as those clocks agree. A zero time means unknown: the item was written before dynago kept timestamps.

func StampsOf 

func StampsOf(raw dynamo.Item) Timestamps

StampsOf reads the timestamps of a stored item.

type WriteOption 

type WriteOption func(*WriteOptions)

WriteOption modifies an update or delete.

func From 

func From(entity any) WriteOption

From makes the write start from an entity the store returned earlier (by Get, a query, or a unique lookup) instead of reading the item again. The write fails with ErrVersionMismatch if the item has changed since that entity was read, so no update can be lost.

Example

From starts a write from an entity the store returned: the write skips its own read, and fails with ErrVersionMismatch if the item changed since, so decisions made on the entity still hold. 

package main

import (
	"context"
	"errors"
	"fmt"
	"time"

	"github.com/nicklanng/dynago"
	"github.com/nicklanng/dynago/examples/toollibrary"
)

// These examples use the store generated from examples/toollibrary/toollibrary.dynago.yaml.
var (
	ctx = context.Background()
	st  *toollibrary.Store
)

func main() {
	key := toollibrary.LoanKey{LibraryID: "lib_1", ToolID: "t_42", LoanID: "ln_9"}
	loan, err := st.Loans.Get(ctx, key)
	if err != nil {
		panic(err)
	}
	if loan.DueAt.Sub(loan.BorrowedAt) > 28*24*time.Hour {
		return // business rule decided on the loaded entity: no more extensions
	}
	err = st.Loans.Extend(ctx, key, toollibrary.LoanExtend{DueAt: loan.DueAt.Add(7 * 24 * time.Hour)}, dynago.From(loan))
	if errors.Is(err, dynago.ErrVersionMismatch) {
		fmt.Println("the loan changed; reload and decide again")
	}
}

func IfVersion 

func IfVersion(v string) WriteOption

IfVersion makes the write fail with ErrVersionMismatch unless the item is still at version v (from the entity's Version method, typically round-tripped through an ETag). An empty v means no check, so a missing If-Match header can be passed straight through.

Writes that change nothing derived stay a single conditional UpdateItem; writes that maintain counters, claims or copies still read the item to compute their changes, but fail instead of retrying when the version differs.

Example

IfVersion carries a version across requests, typically as an ETag, so a stale edit form cannot overwrite a newer change. ReturnVersion hands back the new version for the response. 

package main

import (
	"context"
	"errors"
	"net/http"
	"strconv"
	"strings"

	"github.com/nicklanng/dynago"
	"github.com/nicklanng/dynago/examples/toollibrary"
)

// These examples use the store generated from examples/toollibrary/toollibrary.dynago.yaml.
var st *toollibrary.Store

func main() {
	key := toollibrary.ToolKey{LibraryID: "lib_1", ToolID: "t_42"}
	get := func(w http.ResponseWriter, r *http.Request) {
		tool, _ := st.Tools.Get(r.Context(), key)
		w.Header().Set("ETag", strconv.Quote(tool.Version())) // ETags are quoted (RFC 9110)
	}
	put := func(w http.ResponseWriter, r *http.Request) {
		var version string
		err := st.Tools.EditDetails(r.Context(), key, toollibrary.ToolEditDetails{Name: dynago.Ptr(r.FormValue("name"))},
			dynago.IfVersion(strings.Trim(strings.TrimPrefix(r.Header.Get("If-Match"), "W/"), `"`)), dynago.ReturnVersion(&version))
		if errors.Is(err, dynago.ErrVersionMismatch) {
			http.Error(w, "changed by someone else; reload", http.StatusPreconditionFailed)
			return
		}
		w.Header().Set("ETag", strconv.Quote(version))
	}
	_, _ = get, put
}

func ReturnVersion 

func ReturnVersion(dst *string) WriteOption

ReturnVersion stores the item's new version in dst after a successful write, so a handler can return it as an ETag without reading the item again.

type WriteOptions 

type WriteOptions struct {
	// Version is an expected version from IfVersion, or "" for none.
	Version string
	// From is an entity previously returned by the store, or nil.
	From any
	// NewVersion, if set, receives the item's version after a successful write.
	NewVersion *string
}

WriteOptions are the resolved options of a write. Generated code reads them.

func ApplyOptions 

func ApplyOptions(opts []WriteOption) WriteOptions

ApplyOptions resolves write options.

func (WriteOptions) ExpectedRev 

func (o WriteOptions) ExpectedRev() (int64, error)

ExpectedRev returns the revision the caller expects from IfVersion, 0 for none.

func (WriteOptions) Written 

func (o WriteOptions) Written(rev int64)

Written records a successful write's new revision for ReturnVersion.

Source Files

View all Source files

Directories

Path Synopsis
cmd
dynago command
Command dynago generates typed DynamoDB access code, documentation and infrastructure from a schema file, analyses the design, compares designs for review, and finds DynamoDB calls that bypass the schema.
 Command dynago generates typed DynamoDB access code, documentation and infrastructure from a schema file, analyses the design, compares designs for review, and finds DynamoDB calls that bypass the schema.
Package dynagotest connects tests to DynamoDB Local and gives each test its own table.
 Package dynagotest connects tests to DynamoDB Local and gives each test its own table.
examples
toollibrary
Package toollibrary is the dynago example: a fictional neighbourhood tool library.
 Package toollibrary is the dynago example: a fictional neighbourhood tool library.
toollibrary/cmd/migrate-toollibrary command
Command migrate-toollibrary copies the previous generation of the toollibrary table into the current one.
 Command migrate-toollibrary copies the previous generation of the toollibrary table into the current one.
internal
analysis
Package analysis reasons about a schema as a design: how its items group into partitions, how big and busy each partition gets under the declared workload, what the design guarantees, and which choices are risky.
 Package analysis reasons about a schema as a design: how its items group into partitions, how big and busy each partition gets under the declared workload, what the design guarantees, and which choices are risky.
arch
Package arch captures a schema's analysed design as a plain snapshot (what `dynago check -json` prints), and compares two snapshots as an architecture diff for review.
 Package arch captures a schema's analysed design as a plain snapshot (what `dynago check -json` prints), and compares two snapshots as an architecture diff for review.
cost
Package cost estimates item sizes, capacity units and monthly cost for a schema.
 Package cost estimates item sizes, capacity units and monthly cost for a schema.
e2e/fixture
Package fixture is a test-only store for shapes the example doesn't use.
 Package fixture is a test-only store for shapes the example doesn't use.
e2e/rekey
Package rekey is a test-only store whose second table generation changes an entity's key: people were keyed by id, and are keyed by email.
 Package rekey is a test-only store whose second table generation changes an entity's key: people were keyed by id, and are keyed by email.
gen/docs
Package docs renders a schema as a Markdown data model document with Mermaid diagrams.
 Package docs renders a schema as a Markdown data model document with Mermaid diagrams.
gen/gocode
Package gocode generates the typed Go store for a schema.
 Package gocode generates the typed Go store for a schema.
gen/infra
Package infra renders the table definition as Terraform JSON and as a CreateTable request.
 Package infra renders the table definition as Terraform JSON and as a CreateTable request.
keytmpl
Package keytmpl parses key templates such as "LIB#{libraryId}#TOOL#{toolId}".
 Package keytmpl parses key templates such as "LIB#{libraryId}#TOOL#{toolId}".
lock
Package lock keeps the history of each entity's storage shape, and of the table generations, in a checked-in lock file.
 Package lock keeps the history of each entity's storage shape, and of the table generations, in a checked-in lock file.
schema
Package schema parses a dynago schema file, validates it, and resolves it into the model the generators and the cost report work from.
 Package schema parses a dynago schema file, validates it, and resolves it into the model the generators and the cost report work from.
testdb
Package testdb gives dynago's own tests a DynamoDB: DynamoDB Local by default (through dynagotest), or real tables in an AWS account for the maintainer's release checks.
 Package testdb gives dynago's own tests a DynamoDB: DynamoDB Local by default (through dynagotest), or real tables in an AWS account for the maintainer's release checks.
testdb/sweep command
Command sweep deletes the tables dynago's AWS test runs left behind: tables named dynagotest-… and tagged with a creation time more than an hour ago (-age changes it).
 Command sweep deletes the tables dynago's AWS test runs left behind: tables named dynagotest-… and tagged with a creation time more than an hour ago (-age changes it).
vet
Package vet finds DynamoDB calls made outside generated code: requests through the AWS SDK's DynamoDB client, guregu/dynamo, or dynago's runtime, which bypass the schema's declared reads and writes, and function values that would (`retry(c.PutItem)`).
 Package vet finds DynamoDB calls made outside generated code: requests through the AWS SDK's DynamoDB client, guregu/dynamo, or dynago's runtime, which bypass the schema's declared reads and writes, and function values that would (`retry(c.PutItem)`).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.