dynaflux

package module
v0.0.0-...-de77d72 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

README

Dynaflux

Dynaflux is a Go library and CLI for event-sourced fact streams and materialized views on DynamoDB Global Tables. You declare streams, views and queries in ClickHouse-flavored SQL. Dynaflux rejects, at build time, any query that DynamoDB can't serve efficiently, and estimates what each event and query costs. It generates a typed Go client and the CloudFormation templates that run your store active-active in several AWS regions. None of it has run on AWS yet: see Status.

Why

DynamoDB Global Tables is one of the few managed databases that runs active-active across regions, with little operational work, even during a region outage. Its constraints are hard for developers who come from SQL, though. A common mistake is hand-written code that writes many items per event to serve every lookup.

ClickHouse gives developers a productive model: append-only events, materialized views that update on insert, and pre-registered queries with parameters. But ClickHouse doesn't run active-active across regions. Dynaflux aims to give the ClickHouse model, for the subset DynamoDB serves well, with DynamoDB's multi-region guarantees. Its design follows a few rules:

  • Multi-region first. Every region accepts writes, and the surviving regions keep working during an outage, with no coordination between regions.
  • Pay at write time, not read time. Views do the work as events arrive, so every read is a key lookup or a bounded range read.
  • Events never change. All derived state can be rebuilt from them.
  • A library, not a service. No Dynaflux server sits in the request path, and you own the generated infrastructure.

Every query is declared ahead of time and there are no joins. For ad-hoc questions, Athena reads the archive of every event that Dynaflux keeps in S3.

What you get

  • dynaflux validate checks a project and reports each problem with its file, line and a hint on how to fix it. dynaflux explain shows the table layout and the estimated cost of a stream, view or query.
  • dynaflux generate writes a typed Go client package, the main.go of the Lambda binary that runs the store's functions, and the CloudFormation templates. It numbers view versions in dynaflux.lock, a file you commit.
  • The library appends events and runs queries. DEDUP BY makes appends safe to retry, SYNC views update in the append's transaction, and wait tokens let an application read its own writes.
  • In each region, a Lambda function, the projector, keeps the views current. It also archives every event to Apache Iceberg tables on S3, for Athena and for backfills.
  • The operator commands work on a deployed store: status, diff, backfill, retire, inspect, query and import.
  • dynaflux dev runs the store on DynamoDB Local while you build the application, and fluxtest gives its tests an in-memory store.
  • A query marked -- expose: http is served over HTTP, for clients in other languages.

Install

Dynaflux needs Go 1.26 or later. A project that uses it is a Go module. In the module's directory, add the CLI as a Go tool, so that go.mod pins its version and the library's:

go get -tool github.com/lucindo/dynaflux/cmd/dynaflux@latest
go tool dynaflux version

version shows the versions of the CLI, of the library that go.mod requires, and of the code that generate writes.

Or install the CLI on your PATH. Then dynaflux init tells you to add the library to your module with go get:

go install github.com/lucindo/dynaflux/cmd/dynaflux@latest

Example

This example counts page views. Start in an empty module, without dynaflux init, which writes a similar example of its own. The commands use dynaflux; if you added it as a tool, run go tool dynaflux instead.

At the root of your project, create dynaflux.yaml with the store's name and its regions:

store: analytics
regions: [us-east-1, us-west-2]

Then create dynaflux/pages.sql with a stream of page views, a view with daily totals per page, and a query over that view:

CREATE STREAM page_views (
    user_id     String,
    page        String,
    duration_ms UInt32
)
PRIMARY KEY (user_id);

CREATE MATERIALIZED VIEW page_daily
ORDER BY (page, day)
PRIMARY KEY (page)
AS SELECT
    page,
    toStartOfDay(event_time) AS day,
    count()                  AS views,
    uniqExact(user_id)       AS visitors,
    avg(duration_ms)         AS avg_ms
FROM page_views
GROUP BY page, day;

-- name: PageStats :many
SELECT day, views, visitors, avg_ms
FROM page_daily
WHERE page = {page:String}
  AND day BETWEEN {from:Date} AND {to:Date}
ORDER BY day;
  • The stream's PRIMARY KEY groups events by user. Every event also has implicit columns, such as the event_time the view uses.
  • The view keeps one row per page and day. Its PRIMARY KEY is the DynamoDB partition key, and the rest of ORDER BY sorts rows within a partition.
  • PageStats fixes the partition key and takes a range of days, so it costs one DynamoDB Query on one partition.

Check the project. No output means no problems:

$ dynaflux validate

Ask what the stream and its view cost at 100 events per second:

$ dynaflux explain -rate 100 page_views
stream page_views
  assumptions  -rate 100  -event-kb 1  -rows page_daily=1000  -uniq-new page_daily=0.3
               (traffic isn't in the SQL or dynaflux.yaml; set it with these flags)
  prices       us-east-1 list prices, ±30%: for comparing designs, not a quote

layout, in each of 2 regions (us-east-1, us-west-2)
  table        analytics-stream-page_views: Global Table, on-demand
  key          pk user_id; sk event_time, event_id
  log          LOG RETENTION 2 DAY: ~19 GB in ~2 partitions, each a change-stream shard
  projector    batching window 1 s, batches of at most 100 events: ~2 invocations/s, 50 events each on average
  marks        a view row keeps a shard's high-water mark for 960 s after the shard's progress passes it (runtime.prune_window_seconds)
  archive      Firehose to Iceberg in us-east-1, us-west-2

per event, per region                                     write units  read units
  event put, local or replicated                                 1.00
  TTL delete, replicated from the region that expires it         0.50
  page_daily, shared rows, 1,000 active                          1.07        1.07
    uniqExact markers                                            0.30        1.00
  progress item, one per batch                                   0.02
  total                                                          2.89        2.07
  per second, all events                                          289         207

monthly                        per region  2 regions
  DynamoDB requests                  $543     $1,087
    stream and meta tables           $250       $499
    page_daily                       $294       $587
  DynamoDB storage of the log       $4.75      $9.50
  Lambda, the projector               $10        $20
  Firehose                            $14        $28
  S3 archive, year-1 average        $7.25        $15
  replication transfer              $2.63      $5.26
  total                              $582     $1,165
  provisioned capacity at a 70% target would cut requests to $224 per region (-59%), but costs more than on-demand if it must cover peaks over 2.42 times the average
  not modeled: query reads, view and marker storage, backfills, archive compaction, CloudWatch

dynaflux explain page_daily and dynaflux explain PageStats show the view and the query on their own.

Now add a query that DynamoDB can't serve without scanning every page. Create dynaflux/top_pages.sql:

-- name: TopPages :many
SELECT page, views
FROM page_daily
WHERE day = {day:Date}
ORDER BY views DESC
LIMIT 10;

validate rejects it, and exits with status 1:

$ dynaflux validate
dynaflux/top_pages.sql:4:1: error DF4101: TopPages: view page_daily is partitioned by (page), but WHERE does not bind "page" with = or IN. Every query must fix the partition key.
  hint: add page = {page:String} to WHERE, or declare a view whose PRIMARY KEY matches this query

Getting started builds a store like this one step by step, runs it on your machine, then deploys it to two AWS regions.

Status

Dynaflux is preparing its first release, v1.0.0. Release candidates come first, tagged v1.0.0-rc.1 and up. CHANGELOG.md lists what each release changes.

Everything above is tested without AWS: unit tests, a simulation of three regions with faults, integration tests on DynamoDB Local, and an end-to-end suite that runs the example store on DynamoDB Local. Nothing has run on AWS yet. Before v1.0.0, a release candidate must pass two runs on AWS: the end-to-end suite, and the getting started guide followed from the first command to the teardown.

Until v1.0.0, anything may still change. From v1.0.0 on, every v1.x release keeps the Go API, your projects, the stored data, the AWS resource names, the CLI's commands and flags, and the wire formats compatible, as the compatibility promise lists.

Documentation

The guides:

  • Getting started: build a store, run it on your machine, deploy it to two AWS regions, and delete it.
  • Concepts: the parts of a store, how an event flows, and what to expect across regions and during an outage.
  • Writing the application: the generated Go client, values, errors and tests.
  • SQL recipes: streams, views and queries for common tasks.
  • Operating a store: status, change procedures, backfills, alarms and the archive.
  • What a store costs: where the money goes, the settings that lower it, and how to read explain.

docs/README.md also lists the specifications, which define what Dynaflux accepts and does, and the architecture decision records. examples/analytics is a larger store in two regions, with every generated file committed.

Contributing

CONTRIBUTING.md explains how to build and test Dynaflux, and how to keep the docs' outputs current.

License

Apache License 2.0. See LICENSE.

Documentation

Overview

Package dynaflux runs event-sourced fact streams and materialized views on DynamoDB Global Tables. You declare streams, views and queries in SQL, and the dynaflux command generates typed Go code and infrastructure from them.

An application opens a Store, as Open's example shows, and passes it to the generated package: here flux, the client of a store with a stream page_views and a query PageStats over a view of it:

client := flux.New(store)

An append with Waitable returns a token that a query can wait for, so that the query's rows include the event:

view := flux.PageView{UserID: "u1", Page: "/home", EventTime: time.Now()}
tok, err := client.AppendPageView(ctx, view, dynaflux.Waitable())
if err != nil {
	return err
}
day := flux.PageStatsParams{Page: view.Page, From: view.EventTime, To: view.EventTime}
rows, err := client.PageStats(ctx, day, dynaflux.WaitFor(tok, 5*time.Second))
if err != nil {
	return err
}
for _, row := range rows {
	fmt.Println(row.Day, row.Views, row.Visitors)
}

A stream read is a range loop, which yields at most one error:

key := flux.PageViewsKey{UserID: "u1"}
for ev, err := range client.ReadPageViews(ctx, key, dynaflux.Since(time.Now().Add(-time.Hour))) {
	if err != nil {
		return err
	}
	fmt.Println(ev.Page, ev.EventTime)
}

Errors wrap the sentinels below, for errors.Is, and the types ValueError and BatchError, for errors.AsType:

switch ve, ok := errors.AsType[*dynaflux.ValueError](err); {
case ok:
	fmt.Println("Dynaflux can't store the value of", ve.Name) // a column or query parameter
case errors.Is(err, dynaflux.ErrDuplicate):
	// A stream with DEDUP BY stored the event before; tok names that event.
case errors.Is(err, dynaflux.ErrWaitTimeout):
	// The projector hasn't handled the event yet: query again later.
}

The getting-started guide builds and deploys such an application.

Index

Examples

Constants

View Source
const GeneratedCodeVersion1 = 1

GeneratedCodeVersion1 is the contract between the code that dynaflux generate writes and this module. Generated code names the generator's current GeneratedCodeVersionN, whatever features the project uses. When new generated code needs something that an older module lacks, the generator names the next N, so that code fails to compile at that line against the older module, not with "undefined: …". Every v1 release keeps every constant it has defined.

Variables

View Source
var (
	// ErrDuplicate is an event that another append stored before: the same PRIMARY KEY,
	// event_time and DEDUP BY values. The append returns that event's Token.
	ErrDuplicate = engine.ErrDuplicate // "dynaflux: duplicate event"
	// ErrOutsideWindow is an event_time outside the stream's ACCEPT window. An append with
	// [AsImport] skips the window.
	ErrOutsideWindow = engine.ErrOutsideWindow // "dynaflux: event_time outside the ACCEPT window"
	// ErrInvalidValue is what Dynaflux can't store:
	//   - a value, which a [ValueError] names
	//   - an event over DynamoDB's 400 KB item limit
	//   - an event of an archived stream whose archive record could be over Firehose's
	//     record limit of 1,024,000 bytes; the error names the stream and the size
	ErrInvalidValue = engine.ErrInvalidValue // "invalid value"
	// ErrNotFound is a :one query that found no row.
	ErrNotFound = engine.ErrNotFound // "dynaflux: no row found"
	// ErrTooManyKeys is a query call whose IN parameters make more than 100 key
	// combinations to read.
	ErrTooManyKeys = engine.ErrTooManyKeys // "too many key combinations"
	// ErrEvaluation is an expression of a query that failed on the rows it read, such as an
	// integer division by zero, or a sum outside its result type; or one of a SYNC view
	// that failed on an event appended without [AsImport], which then isn't written. The
	// error names the view version.
	ErrEvaluation = engine.ErrEvaluation // "evaluation error"
	// ErrViewNotReady is a query over a view whose version isn't live in the store's region:
	// no backfill has made it live yet, or it broke or was retired. The error names the
	// version, its state and the region, and a broken version's failure. [AllowPartial]
	// reads the view anyway.
	ErrViewNotReady = engine.ErrViewNotReady // "view version not ready"
	// ErrWaitTimeout is a query whose [WaitFor] timeout passed before the projector had
	// handled every event of the token.
	ErrWaitTimeout = engine.ErrWaitTimeout // "wait for an append timed out"
)

Errors to check with errors.Is. Returned errors wrap them with details.

Functions

This section is empty.

Types

type AppendOption

type AppendOption interface {
	// contains filtered or unexported methods
}

AppendOption configures an append. Every generated append method takes them.

func AsImport

func AsImport() AppendOption

AsImport appends past events as an import: the append skips the stream's ACCEPT window, and it updates no SYNC view in its transaction, so a SYNC view gives no read-your-writes for them. The projector applies the events to every view, as it does for ASYNC views: an event that a SYNC view can't evaluate is stored anyway, and breaks that view version, as it would an ASYNC one. Combine it with Waitable to wait for them. Into a stream without DEDUP BY, a retried import stores the events again.

func Waitable

func Waitable() AppendOption

Waitable asks the projector of the store's region to write a receipt for each event of the append once it has handled it, so that WaitFor can wait for them: one more write to the store's meta table per event, which other appends don't pay.

type BatchError

type BatchError struct {
	Errs []error
}

BatchError is the error of a batch append. Errs lines up with the events, nil for each one the call stored.

Example

A retried batch on a stream with DEDUP BY reports the events that an earlier try stored as ErrDuplicate. errors.Is on the whole error can't tell those from failures, so check each event. Here the retry stored event 0, an earlier try stored event 1, and event 2 isn't stored.

package main

import (
	"errors"
	"fmt"

	"github.com/lucindo/dynaflux"
)

func main() {
	err := error(&dynaflux.BatchError{Errs: []error{
		nil,
		dynaflux.ErrDuplicate,
		&dynaflux.ValueError{Name: "price", Err: fmt.Errorf("%w: more digits than its scale", dynaflux.ErrInvalidValue)},
	}})
	if be, ok := errors.AsType[*dynaflux.BatchError](err); ok {
		for i, e := range be.Errs {
			if e != nil && !errors.Is(e, dynaflux.ErrDuplicate) {
				fmt.Printf("event %d was not stored: %v\n", i, e)
			}
		}
	}
}
Output:
event 2 was not stored: dynaflux: price: invalid value: more digits than its scale

func (*BatchError) Error

func (e *BatchError) Error() string

func (*BatchError) Unwrap

func (e *BatchError) Unwrap() []error

Unwrap returns the errors of the events that failed.

type Catalog

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog holds a store's definitions, compiled. The generated package declares one, as Catalog in its catalog.go. The zero Catalog can't be used.

type Config

type Config struct {
	// AWS configures the DynamoDB client. Its Region is the region the store appends in,
	// which events record as origin_region; it must be one of the regions in dynaflux.yaml.
	// Dynaflux uses its credentials, retryer, HTTP client and endpoint as they are.
	//
	// The projector's client is a model to follow:
	//   - the standard retryer without its client-side retry quota (ratelimit.None)
	//   - a 10 s HTTP timeout for each attempt
	AWS aws.Config
	// Prefix is the deployment's prefix of resource names: the templates' Prefix
	// parameter, "" for none.
	Prefix string
}

Config configures the Store that Open returns.

type Decimal

type Decimal struct {
	// contains filtered or unexported fields
}

Decimal is an exact decimal number of at most 38 digits that keeps its scale, the digits after its point: 12.50 has scale 2. The zero value is 0.

== compares the representation, scale included, so 12.5 and 12.50 differ under == and are Decimal.Equal. Values read from one Decimal(P, S) column all have scale S.

func DecimalFromBig

func DecimalFromBig(unscaled *big.Int, scale int) (Decimal, error)

DecimalFromBig returns unscaled × 10^−scale. It fails when unscaled has more than 38 digits or scale is outside 0 to 38.

func ParseDecimal

func ParseDecimal(s string) (Decimal, error)

ParseDecimal parses an optional '-', digits, and optionally '.' and more digits, such as "-12.50". The scale is the number of digits after the point.

Example

A Decimal keeps its scale: == compares the representation, and Equal the value.

package main

import (
	"fmt"

	"github.com/lucindo/dynaflux"
)

func main() {
	a, err := dynaflux.ParseDecimal("12.5")
	if err != nil {
		fmt.Println(err)
		return
	}
	b, err := dynaflux.ParseDecimal("12.50")
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(a, b)
	fmt.Println(a == b, a.Equal(b))
}
Output:
12.5 12.50
false true

func (Decimal) Big

func (d Decimal) Big() (*big.Int, int)

Big returns d's unscaled value and scale: d is unscaled × 10^−scale.

func (Decimal) Compare

func (d Decimal) Compare(e Decimal) int

Compare returns -1, 0 or +1 as d is less than, equal to or greater than e, by value.

func (Decimal) Equal

func (d Decimal) Equal(e Decimal) bool

Equal reports whether d and e have the same value, whatever their scales.

func (Decimal) MarshalText

func (d Decimal) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler with Decimal.String.

func (Decimal) String

func (d Decimal) String() string

String returns d with exactly its scale's digits after the point, such as "-0.05".

func (*Decimal) UnmarshalText

func (d *Decimal) UnmarshalText(b []byte) error

UnmarshalText implements encoding.TextUnmarshaler with ParseDecimal.

type QueryOption

type QueryOption interface {
	// contains filtered or unexported methods
}

QueryOption configures a query.

func AllowPartial

func AllowPartial() QueryOption

AllowPartial runs a query over a view whose version isn't live in the store's region, without reading its state, instead of returning ErrViewNotReady. The rows may miss events: those a backfill hasn't applied yet, or every event after the version broke.

func WaitFor

func WaitFor(tok Token, timeout time.Duration) QueryOption

WaitFor makes a query first wait until the projector of the store's region has handled every event of tok, an append's token in this region, for at most timeout. A query over a view then reads its version's state again, and both kinds read strongly consistently, so the rows include the events. A wait takes about the projector's batching window plus one invocation, 1 to 1.5 seconds by default: give it a timeout of a few seconds. When the timeout passes first, the query returns an error that matches ErrWaitTimeout; when ctx ends first, ctx's error.

The query fails at once, before any read:

  • in another region than the append's
  • for an event appended without Waitable
  • for an event that another region stored, as a duplicate of a replicated DEDUP BY event is
  • for a token of more than 1,000 events
  • for a token of a stream that the store doesn't have

A token can be waited on for 24 hours after its append. The zero Token waits for nothing. Given more than once, WaitFor makes the query wait for each token in turn, each for its own timeout.

type ReadOption

type ReadOption interface {
	// contains filtered or unexported methods
}

ReadOption configures a stream read.

func Since

func Since(t time.Time) ReadOption

Since reads the events whose event_time is t or later.

func Until

func Until(t time.Time) ReadOption

Until reads the events whose event_time is before t. With Since, windows that share an end chain without a gap or a repeat.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store connects generated clients to every Dynaflux store deployed with Config.Prefix in the region of Config.AWS. In SQL and in errors, a store is the catalog of one generated package, such as store analytics; one Store serves all of them. Open returns one backed by DynamoDB; in tests, fluxtest.NewStore makes one in memory. A Store is safe for concurrent use. The zero Store can't be used.

func Open

func Open(cfg Config) (*Store, error)

Open returns a store backed by the DynamoDB tables that the generated templates create for cfg.Prefix in cfg.AWS.Region, which must be a region code such as us-east-1: for FIPS endpoints, set AWS_USE_FIPS_ENDPOINT instead of naming a FIPS pseudo-region. It makes no AWS call, so a missing table or grant shows on the first append, read or query.

A query over a view checks that the view's version is live in the region, in the view states it read from the store's meta table at most 30 seconds before, and returns ErrViewNotReady when it isn't. The application needs dynamodb:Query on the meta table for that read.

Example

An application's Store, with the AWS client that Config.AWS recommends. The prefix is the one dynaflux dev prints, or the one the templates were deployed with.

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/aws/ratelimit"
	"github.com/aws/aws-sdk-go-v2/aws/retry"

	awshttp "github.com/aws/aws-sdk-go-v2/aws/transport/http"
	"github.com/aws/aws-sdk-go-v2/config"

	"github.com/lucindo/dynaflux"
)

func main() {
	ctx := context.Background()
	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRetryer(func() aws.Retryer {
			return retry.NewStandard(func(o *retry.StandardOptions) { o.RateLimiter = ratelimit.None })
		}),
		config.WithHTTPClient(awshttp.NewBuildableClient().WithTimeout(10*time.Second)))
	if err != nil {
		fmt.Println(err)
		return
	}
	store, err := dynaflux.Open(dynaflux.Config{AWS: cfg, Prefix: os.Getenv("DYNAFLUX_PREFIX")})
	if err != nil {
		fmt.Println(err)
		return
	}
	_ = store // pass it to the generated package's New
}

type Token

type Token struct {
	// contains filtered or unexported fields
}

Token names the events that an append stored, in the store's region. The zero Token names none.

func ParseToken

func ParseToken(s string) (Token, error)

ParseToken returns the Token whose text form is s, so that another process in the same region can wait for its events with WaitFor. It checks only the form: a token is unauthenticated text that grants nothing, since WaitFor reads only receipts with it.

The text form is a wire format. Every release reads the token formats of the releases before it, and a release writes a new format only after a release that reads it has shipped. So during a rolling deploy from the previous release, each process reads the tokens of the others.

func (Token) IsZero

func (t Token) IsZero() bool

IsZero reports whether t is the zero Token.

func (Token) MarshalText

func (t Token) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler with Token.String; the zero Token is "".

Example

A Token in a JSON document is its text, and the zero Token is "".

package main

import (
	"encoding/json"
	"fmt"

	"github.com/lucindo/dynaflux"
)

func main() {
	tok, err := dynaflux.ParseToken("AQVsb2NhbApwYWdlX3ZpZXdzANxqz6wA2Yd-zm02iqwab4DA39mToNcBAA")
	if err != nil {
		fmt.Println(err)
		return
	}
	type receipt struct {
		Token dynaflux.Token `json:"wait"`
		Prior dynaflux.Token `json:"prior"`
	}
	data, err := json.Marshal(receipt{Token: tok})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(string(data))
	var back receipt
	if err = json.Unmarshal(data, &back); err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(back.Token == tok, back.Prior.IsZero())
}
Output:
{"wait":"AQVsb2NhbApwYWdlX3ZpZXdzANxqz6wA2Yd-zm02iqwab4DA39mToNcBAA","prior":""}
true true

func (Token) String

func (t Token) String() string

String returns the token's text form, which is opaque. ParseToken reads it back.

func (*Token) UnmarshalText

func (t *Token) UnmarshalText(b []byte) error

UnmarshalText implements encoding.TextUnmarshaler with ParseToken; "" is the zero Token.

type UUID

type UUID [16]byte

UUID is a 128-bit UUID. Its methods behave as those of Go 1.27's uuid.UUID, and the two types convert into each other for free. UUID stays a [16]byte type of its own for all of v1. To make one, read crypto/rand into it, rand.Read(id[:]), or on Go 1.27 convert uuid.NewV7: dynaflux.UUID(uuid.NewV7()).

func ParseUUID

func ParseUUID(s string) (UUID, error)

ParseUUID parses a UUID in one of the forms Go 1.27's uuid.Parse accepts, with hex digits in either case:

f81d4fae-7dec-11d0-a765-00a0c91e6bf6
{f81d4fae-7dec-11d0-a765-00a0c91e6bf6}
urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6
f81d4fae7dec11d0a76500a0c91e6bf6
Example

ParseUUID accepts four forms, and String prints one.

package main

import (
	"fmt"

	"github.com/lucindo/dynaflux"
)

func main() {
	for _, s := range []string{
		"f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
		"{F81D4FAE-7DEC-11D0-A765-00A0C91E6BF6}",
		"urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
		"f81d4fae7dec11d0a76500a0c91e6bf6",
	} {
		u, err := dynaflux.ParseUUID(s)
		if err != nil {
			fmt.Println(err)
			return
		}
		fmt.Println(u)
	}
}
Output:
f81d4fae-7dec-11d0-a765-00a0c91e6bf6
f81d4fae-7dec-11d0-a765-00a0c91e6bf6
f81d4fae-7dec-11d0-a765-00a0c91e6bf6
f81d4fae-7dec-11d0-a765-00a0c91e6bf6

func (UUID) AppendText

func (u UUID) AppendText(b []byte) ([]byte, error)

AppendText implements encoding.TextAppender with the form of UUID.String.

func (UUID) Compare

func (u UUID) Compare(v UUID) int

Compare returns -1, 0 or +1 as u sorts before, with or after v, byte by byte.

func (UUID) MarshalText

func (u UUID) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler with the form of UUID.String.

func (UUID) String

func (u UUID) String() string

String returns u in lowercase 8-4-4-4-12 form.

func (*UUID) UnmarshalText

func (u *UUID) UnmarshalText(b []byte) error

UnmarshalText implements encoding.TextUnmarshaler; it accepts the forms of ParseUUID.

type ValueError

type ValueError struct {
	// Name is the column or the query parameter, as in SQL; for a PRIMARY KEY too long for
	// DynamoDB, the key's columns joined by ", ", such as "org_id, user_id".
	Name string
	Err  error
}

ValueError is a value that Dynaflux can't store: invalid UTF-8, a Float64 that isn't finite or is outside DynamoDB's range, a Decimal outside its type, a time outside 1970 to 2106, an array of more than 1,000 elements, a PRIMARY KEY longer than DynamoDB allows, or a missing event_time on a stream with DEDUP BY. Err matches ErrInvalidValue.

func (*ValueError) Error

func (e *ValueError) Error() string

func (*ValueError) Unwrap

func (e *ValueError) Unwrap() error

Directories

Path Synopsis
cmd
dynaflux command
Dynaflux checks a Dynaflux project's SQL definitions and dynaflux.yaml, explains the layout and cost of its streams, views and queries, generates its typed Go client package and dynaflux.lock, and reads and operates the deployed store.
Dynaflux checks a Dynaflux project's SQL definitions and dynaflux.yaml, explains the layout and cost of its streams, views and queries, generates its typed Go client package and dynaflux.lock, and reads and operates the deployed store.
examples
analytics
Package analytics is application code that uses the example's generated client.
Package analytics is application code that uses the example's generated client.
analytics/cmd/dynaflux-runtime command
Command dynaflux-runtime is the Lambda binary of store analytics.
Command dynaflux-runtime is the Lambda binary of store analytics.
analytics/internal/flux
Package flux is the typed client of store analytics, generated by dynaflux from its SQL definitions.
Package flux is the typed client of store analytics, generated by dynaflux from its SQL definitions.
Package fluxgen holds the functions that code generated by dynaflux generate calls.
Package fluxgen holds the functions that code generated by dynaflux generate calls.
Package fluxruntime is the entry point of the Lambda binary that dynaflux generate writes to runtime.out.
Package fluxruntime is the entry point of the Lambda binary that dynaflux generate writes to runtime.out.
Package fluxtest runs a Dynaflux store in memory, for tests of code that uses a generated client.
Package fluxtest runs a Dynaflux store in memory, for tests of code that uses a generated client.
internal
access
Package access lets fluxgen, fluxtest and fluxruntime build and open the types of package dynaflux, which has no exported constructor that takes internal types.
Package access lets fluxgen, fluxtest and fluxruntime build and open the types of package dynaflux, which has no exported constructor that takes internal types.
agg
Package agg implements the aggregate states of docs/spec/encoding.md §7.2–§7.5: what a view stores for an aggregate column, how inputs and other states merge into it, its finalized value, and the largest attribute it can need (§7.8).
Package agg implements the aggregate states of docs/spec/encoding.md §7.2–§7.5: what a view stores for an aggregate column, how inputs and other states merge into it, its finalized value, and the largest attribute it can need (§7.8).
archive
Package archive is the schema of the event archive: in each archiving region, an Iceberg table per archived stream, which Firehose writes, and an Athena view of it with one row per event.
Package archive is the schema of the event archive: in each archiving region, an Iceberg table per archived stream, which Firehose writes, and an Athena view of it with one row per event.
archive/athenafail
Package athenafail classifies Athena's failures by what a caller does about them, for the setup function's statements and the backfill's exports (ADR 0016).
Package athenafail classifies Athena's failures by what a caller does about them, for the setup function's statements and the backfill's exports (ADR 0016).
archive/firehose
Package firehose writes a projector batch's INSERT records to the stream's Firehose stream, which feeds the region's Iceberg table.
Package firehose writes a projector batch's INSERT records to the stream's Firehose stream, which feeds the region's Iceberg table.
backfill
Package backfill fills one view version in one region (docs/spec/lifecycle.md §5, ADR 0011).
Package backfill fills one view version in one region (docs/spec/lifecycle.md §5, ADR 0011).
backfill/awsbackfill
Package awsbackfill implements the interfaces of internal/backfill with the AWS SDK, for the backfill function of the runtime binary and for dynaflux backfill: staged runs in the region's bucket, a stream's shards, the replicas of a stream table, the projectors' lag, and the archive with what AWS shows of its streams' history.
Package awsbackfill implements the interfaces of internal/backfill with the AWS SDK, for the backfill function of the runtime binary and for dynaflux backfill: staged runs in the region's bucket, a stream's shards, the replicas of a stream table, the projectors' lag, and the archive with what AWS shows of its streams' history.
backfill/awsbackfill/awsbackfilltest
Package awsbackfilltest serves, over HTTP, the AWS calls of package awsbackfill that the emulator lacks, for tests: S3 objects in memory, Athena running package archive's statements over records in memory, CloudWatch with series a test sets or without data, Firehose's description of a delivery stream, and Lambda's event source mappings.
Package awsbackfilltest serves, over HTTP, the AWS calls of package awsbackfill that the emulator lacks, for tests: S3 objects in memory, Athena running package archive's statements over records in memory, CloudWatch with series a test sets or without data, Firehose's description of a delivery stream, and Lambda's event source mappings.
catalog
Package catalog is the compiled model of a Dynaflux store: its streams, every live version of its views, and its named queries, with expressions as a typed IR.
Package catalog is the compiled model of a Dynaflux store: its streams, every live version of its views, and its named queries, with expressions as a typed IR.
catalog/catalogtest
Package catalogtest builds catalogs from SQL for tests.
Package catalogtest builds catalogs from SQL for tests.
cli
Package cli implements the dynaflux command.
Package cli implements the dynaflux command.
codegen
Package codegen writes the typed Go client that dynaflux generate puts in go.out, and the main.go of the runtime binary that it puts in runtime.out.
Package codegen writes the typed Go client that dynaflux generate puts in go.out, and the main.go of the runtime binary that it puts in runtime.out.
config
Package config loads dynaflux.yaml, the configuration of one Dynaflux store, as docs/spec/config.md specifies.
Package config loads dynaflux.yaml, the configuration of one Dynaflux store, as docs/spec/config.md specifies.
deployed
Package deployed reads one region of a deployed store for the operator commands: its registered catalog, and from the services around the meta table the backfill jobs' executions, the event source mappings, the archive's metrics, and the gaps that Lambda recorded in the store's bucket.
Package deployed reads one region of a deployed store for the operator commands: its registered catalog, and from the services around the meta table the backfill jobs' executions, the event source mappings, the archive's metrics, and the gaps that Lambda recorded in the store's bucket.
diag
Package diag holds the diagnostics the Dynaflux front end reports for SQL definitions, dynaflux.yaml and dynaflux.lock, in the format of docs/spec/sql.md appendix A.1.
Package diag holds the diagnostics the Dynaflux front end reports for SQL definitions, dynaflux.yaml and dynaflux.lock, in the format of docs/spec/sql.md appendix A.1.
emf
Package emf publishes counts in a function's log line as CloudWatch metrics, in the embedded metric format, under the namespace and dimension that the generated alarms read.
Package emf publishes counts in a function's log line as CloudWatch metrics, in the embedded metric format, under the namespace and dimension that the generated alarms read.
emulatortest
Package emulatortest connects tests to DynamoDB Local (docs/decisions/0009-emulator-choice.md): clients that can reach only a loopback endpoint, tables of their own for each test, and a recorder of every request a client sends.
Package emulatortest connects tests to DynamoDB Local (docs/decisions/0009-emulator-choice.md): clients that can reach only a loopback endpoint, tables of their own for each test, and a recorder of every request a client sends.
engine
Package engine appends to and reads streams, runs named queries and reads view rows over a store.Backend, with the rules of sql.md §4.3, §8 and §10, for every way into a store: generated code, the CLI and the query handler.
Package engine appends to and reads streams, runs named queries and reads view rows over a store.Backend, with the rules of sql.md §4.3, §8 and §10, for every way into a store: generated code, the CLI and the query handler.
esmlocal
Package esmlocal reads a DynamoDB stream on an emulator and renders its records as the tumbling-window events Lambda's event source mapping sends (docs/decisions/0003).
Package esmlocal reads a DynamoDB stream on an emulator and renders its records as the tumbling-window events Lambda's event source mapping sends (docs/decisions/0003).
explain
Package explain estimates what a stream and its views cost to run: request units, row sizes, projector invocations and monthly dollars, per region.
Package explain estimates what a stream and its views cost to run: request units, row sizes, projector invocations and monthly dollars, per region.
expr
Package expr evaluates the catalog's expressions over SQL values.
Package expr evaluates the catalog's expressions over SQL values.
fanout
Package fanout runs a function over indexes on goroutines, a bounded number at a time.
Package fanout runs a function over indexes on goroutines, a bounded number at a time.
goname
Package goname derives the Go names that dynaflux generate declares (sql.md §3.2).
Package goname derives the Go names that dynaflux generate declares (sql.md §3.2).
gotype
Package gotype maps SQL types to the Go types of generated code, for code generation and for package fluxgen alike.
Package gotype maps SQL types to the Go types of generated code, for code generation and for package fluxgen alike.
infra
Package infra builds the infrastructure manifest: the description of one store's AWS resources that dynaflux generate derives from dynaflux.yaml and the catalog, and that a renderer such as package cloudformation turns into templates.
Package infra builds the infrastructure manifest: the description of one store's AWS resources that dynaflux generate derives from dynaflux.yaml and the catalog, and that a renderer such as package cloudformation turns into templates.
infra/cloudformation
Package cloudformation renders an infrastructure manifest as CloudFormation templates, which docs/spec/infra.md describes.
Package cloudformation renders an infrastructure manifest as CloudFormation templates, which docs/spec/infra.md describes.
infra/infratest
Package infratest evaluates a manifest's IAM grants for tests, as IAM would for a request.
Package infratest evaluates a manifest's IAM grants for tests, as IAM would for a request.
keys
Package keys implements the storage encoding of docs/spec/encoding.md: item keys and sort-key ranges, hashes and identifiers, attribute values, and high-water marks.
Package keys implements the storage encoding of docs/spec/encoding.md: item keys and sort-key ranges, hashes and identifiers, attribute values, and high-water marks.
localstore
Package localstore creates and deletes a store's tables on DynamoDB Local as the manifest of internal/infra declares them: keys, on-demand billing, the stream view and the TTL attribute, so a disagreement between the templates and the runtime shows locally.
Package localstore creates and deletes a store's tables on DynamoDB Local as the manifest of internal/infra declares them: keys, on-demand billing, the stream view and the TTL attribute, so a disagreement between the templates and the runtime shows locally.
meta
Package meta reads and writes the items of a store's regional meta table (docs/spec/encoding.md §8): one unit item per delivery unit, with its progress and fences; one view-state item per view version; one job item per backfill job; and the deployed catalog.
Package meta reads and writes the items of a store's regional meta table (docs/spec/encoding.md §8): one unit item per delivery unit, with its progress and fences; one view-state item per view version; one job item per backfill job; and the deployed catalog.
projector
Package projector keeps views current: it applies a delivery unit's change-stream records to every view version of their stream, with conditional writes that make a redelivered record count once (docs/spec/encoding.md §7).
Package projector keeps views current: it applies a delivery unit's change-stream records to every view version of their stream, with conditional writes that make a redelivered record count once (docs/spec/encoding.md §7).
projector/projectortest
Package projectortest selects a backfill's archive records the simple way, as the oracle of tests of the backfill.
Package projectortest selects a backfill's archive records the simple way, as the oracle of tests of the backfill.
queryhttp
Package queryhttp serves a store's exposed named queries, those annotated -- expose: http, to Lambda function URLs, API Gateway HTTP and REST APIs, and direct invoke (ADR 0018).
Package queryhttp serves a store's exposed named queries, those annotated -- expose: http, to Lambda function URLs, API Gateway HTTP and REST APIs, and direct invoke (ADR 0018).
setup
Package setup answers the CloudFormation custom resources of regional.yaml, which the runtime binary serves in its setup mode (ADR 0008, docs/spec/infra.md §6.2, §7.2):
Package setup answers the CloudFormation custom resources of regional.yaml, which the runtime binary serves in its setup mode (ADR 0008, docs/spec/infra.md §6.2, §7.2):
setup/awssetup
Package awssetup implements the interfaces of internal/setup with the AWS SDK, for the setup function of the runtime binary (ADR 0008).
Package awssetup implements the interfaces of internal/setup with the AWS SDK, for the setup function of the runtime binary (ADR 0008).
simulation
Package simulation is the deterministic harness for the projector: regions, change streams, Lambda's event source mapping, faults and an oracle, all in one testing/synctest bubble per seed, so a failing seed replays exactly.
Package simulation is the deterministic harness for the projector: regions, change streams, Lambda's event source mapping, faults and an oracle, all in one testing/synctest bubble per seed, so a failing seed replays exactly.
simulation/mutants command
Command mutants runs the simulation harness's negative controls, several at once.
Command mutants runs the simulation harness's negative controls, several at once.
sql
Package sql parses Dynaflux SQL files into a syntax tree that keeps the shape and positions of the source.
Package sql parses Dynaflux SQL files into a syntax tree that keeps the shape and positions of the source.
store
Package store is the storage boundary: DynamoDB in production, an in-memory store in tests.
Package store is the storage boundary: DynamoDB in production, an in-memory store in tests.
store/dynamo
Package dynamo is the DynamoDB store.Backend.
Package dynamo is the DynamoDB store.Backend.
store/mem
Package mem is an in-memory store.Backend with the DynamoDB behavior Dynaflux depends on: sort keys in byte order, conditions that return the stored item, size limits, numbers stored in DynamoDB's form, and the requests DynamoDB rejects.
Package mem is an in-memory store.Backend with the DynamoDB behavior Dynaflux depends on: sort keys in byte order, conditions that return the stored item, size limits, numbers stored in DynamoDB's form, and the requests DynamoDB rejects.
store/storetest
Package storetest holds the conformance suite that every store.Backend passes, and two faults: an SDK retry after a lost response, and eventually consistent reads that lag.
Package storetest holds the conformance suite that every store.Backend passes, and two faults: an SDK retry after a lost response, and eventually consistent reads that lag.
streamevent
Package streamevent decodes the events of a DynamoDB stream with a tumbling window (ADR 0003), as Lambda delivers them to a projector function and esmlocal does in dynaflux dev, and holds the function's reply.
Package streamevent decodes the events of a DynamoDB stream with a tumbling window (ADR 0003), as Lambda delivers them to a projector function and esmlocal does in dynaflux dev, and holds the function's reply.
txtar
Package txtar reads and writes txtar archives, with the semantics of golang.org/x/tools/txtar: a free-form comment, then files that each start with a marker line "-- name --".
Package txtar reads and writes txtar archives, with the semantics of golang.org/x/tools/txtar: a free-form comment, then files that each start with a marker line "-- name --".
validate
Package validate is the compiler front end of Dynaflux.
Package validate is the compiler front end of Dynaflux.
valuetext
Package valuetext reads and writes values of catalog types outside DynamoDB: as text, for command-line flags and tables, and as JSON, for -json output, import and the HTTP query endpoints.
Package valuetext reads and writes values of catalog types outside DynamoDB: as text, for command-line flags and tables, and as JSON, for -json output, import and the HTTP query endpoints.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL