sink

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 1, 2026 License: MIT Imports: 18 Imported by: 0

README

sink-go

sink-go is the typed, concurrency-safe Go client for the liran/sink gRPC service. It covers the full batch API: reads, puts, self-contained Lua merges, and hard deletes with synchronous or durable asynchronous completion.

Install

go get github.com/liran/sink-go

The client requires Go 1.27 or newer.

Quick start

Dial uses TLS 1.2 or newer by default. The explicit insecure credentials in this example are appropriate only for a trusted local endpoint.

package main

import (
	"context"
	"log"
	"time"

	sink "github.com/liran/sink-go"
	"google.golang.org/grpc/credentials/insecure"
)

type Product struct {
	UID       string    `json:"uid" bson:"_id"`
	Name      string    `json:"name" bson:"name"`
	Stock     int       `json:"stock" bson:"stock"`
	UpdatedAt time.Time `json:"updated_at" bson:"updated_at"`
}

func main() {
	dialOptions := sink.DialOptions{
		TransportCredentials: insecure.NewCredentials(),
	}
	client, err := sink.Dial("127.0.0.1:8080", dialOptions)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	datasetOptions := sink.DatasetOptions{
		Store:     "primary",
		Namespace: "catalog",
		Dataset:   "products",
		Encoding:  sink.DocumentEncodingBSON,
	}
	products, err := sink.NewDataset(client, datasetOptions)
	if err != nil {
		log.Fatal(err)
	}
	value := Product{
		UID:       "product-42",
		Name:      "keyboard",
		Stock:     12,
		UpdatedAt: time.Now().UTC(),
	}
	record := sink.Record{
		Key:   sink.StringKey(value.UID),
		Value: value,
	}
	_, err = products.Upsert(
		context.Background(),
		sink.CompletionWaitUntilVisible,
		record,
	)
	if err != nil {
		log.Fatal(err)
	}
	readResults, err := products.Read(context.Background(), sink.StringKey(value.UID))
	if err != nil {
		log.Fatal(err)
	}
	if readResults[0].Status == sink.ReadFound {
		var stored Product
		if err := readResults[0].Document.Decode(&stored); err != nil {
			log.Fatal(err)
		}
	}
}

Dataset binds the stable store, namespace, dataset, and document encoding once. Read accepts one or many keys without reconstructing addresses. Each mutation still receives an explicit completion mode because callers of the same dataset can require different durability or visibility guarantees. Pass any number of records to Create, Replace, or Upsert; the client validates and encodes the complete collection before sending it, automatically splits large collections by ClientOptions.MaxOperations, and preserves global operation indexes:

records := []sink.Record{
	{Key: sink.StringKey("product-42"), Value: firstProduct},
	{Key: sink.StringKey("product-43"), Value: secondProduct},
}
results, err := products.Upsert(
	context.Background(),
	sink.CompletionReturnAfterAccepted,
	records...,
)

Configure DocumentEncodingBSON for MongoDB; it applies bson tags and keeps native values such as BSON datetimes. Configure DocumentEncodingJSON for Elasticsearch and OpenSearch; it applies json tags and produces ordinary JSON without Extended JSON $date wrappers. Sink rejects an encoding that does not match the selected backend. The low-level API also requires callers to create an explicitly encoded Document before NewPut or NewMerge.

Reads return a Document with Encoding, immutable Payload, and Decode methods. Decode selects the matching JSON or BSON decoder automatically.

A merge rule is ordinary application source code. Construct its immutable LuaProgram once and bind it to the Dataset; every Merge call reuses that program while the client includes its SHA-256 digest automatically:

source := []byte(`
return function(current, incoming)
    current = current or json.object()
    current.stock = incoming.stock
    current.updated_at = sink.v1.time.now()
    return current
end`)
program, err := sink.NewLuaProgram(source)
if err != nil {
	log.Fatal(err)
}
datasetOptions := sink.DatasetOptions{
	Store:        "search-main",
	Namespace:    "catalog",
	Dataset:      "products",
	Encoding:     sink.DocumentEncodingJSON,
	MergeProgram: &program,
}
products, err := sink.NewDataset(client, datasetOptions)
if err != nil {
	log.Fatal(err)
}
record := sink.Record{
	Key:   sink.StringKey("product-42"),
	Value: incomingProduct,
}
results, err := products.Merge(
	context.Background(),
	sink.CompletionWaitUntilVisible,
	sink.MissingDocumentCreate,
	record,
)
if err != nil {
	log.Fatal(err)
}

The stored current document and incoming records must use the Dataset's encoding, and the merge result preserves it. A Dataset without MergeProgram rejects Merge before sending an RPC.

The merge function receives only current and incoming. Sink provides versioned sink.v1 array, object, and retry-stable time helpers. See the Lua merge developer guide for the complete function reference and reliability rules.

API model

  • Dataset is the primary record API. It binds routing, encoding, and one optional Lua program. Read accepts one or many keys; Create, Replace, Upsert, and Merge accept one or many Record values while keeping completion mode explicit per mutation. Every method splits large collections automatically and returns all per-record results.
  • Read(ctx, addresses...) preserves request order and reports found, not-found, or failed results independently.
  • Write(ctx, completionMode, operations...) supports mixed put and merge batches. Use NewPut and NewMerge to construct validated operations.
  • Delete(ctx, completionMode, addresses...) performs hard deletes; deleting an absent record is successful.
  • String, int64, byte, and opaque legacy keys are supported.
  • CheckHealth uses the standard gRPC health service.
  • Raw exposes the generated api/sink/v1 client for advanced use.

Use CompletionWaitUntilApplied for storage acknowledgement, CompletionWaitUntilVisible when a following search read must observe the mutation, and CompletionReturnAfterAccepted for durable asynchronous queueing. For OpenSearch and Elasticsearch, visible completion maps to refresh=wait_for; MongoDB is immediately visible after an acknowledged write.

Lua source travels with the merge intent in both synchronous and asynchronous mode. Sink caches compilation by digest while using a fresh VM per execution, so rules follow application releases without server-side profile management. The client declares identical source only once per Write batch; Sink embeds the full program into each asynchronous Kafka mutation so it remains replayable without server-side rule state.

Each result includes its operation index. The client validates result counts, indexes, statuses, documents, and failure details before returning a response. Per-record failures are represented by OperationError, so one bad record does not hide successful records from the same batch. Dataset methods return all available results and automatically expose operation failures as an errors.As-compatible BatchError. If a later split batch also has a transport failure, the returned error preserves both the earlier operation failures and the transport error. Low-level results expose Err(), and ReadResultsError, WriteResultsError, and DeleteResultsError collect failures explicitly.

Reliability behavior

Reads retry transport-level Unavailable failures and retryable per-operation failures with bounded exponential backoff and jitter. Only failed operations are resubmitted after a partial batch response. The default is three attempts, starting at 100 ms and capped at one second; ClientOptions.ReadRetry can tune or disable retries by setting MaxAttempts to one.

Writes and deletes are never retried automatically. A transport error can arrive after Sink has already applied a synchronous mutation or durably accepted an asynchronous one, so automatic mutation retries could duplicate work. Callers should retry only when their operation is safe under Sink's documented at-least-once semantics.

Read, Write, and Delete split collections into batches of 1,000 operations by default, matching Sink's default configuration. There are no separate All variants for callers to choose between. Set ClientOptions.MaxOperations when the server is configured with a different limit. Encoded requests and responses are limited to 64 MiB by default; use MaxSendMessageBytes and MaxReceiveMessageBytes to match custom server limits. A Client and its underlying gRPC connection are safe for concurrent use.

Compatibility and development

The generated protocol matches the current Sink server contract. CI runs descriptor contract tests, race-enabled unit tests against an in-memory gRPC server, malformed-response tests, static analysis, and an end-to-end compatibility test against the current Sink main branch with MongoDB and Kafka. The compatibility workflow also runs weekly so server-side drift is detected without requiring a client commit.

make test
make test-race
make lint
make proto-check

To run the external compatibility test against an already running server:

SINK_INTEGRATION_ADDRESS=127.0.0.1:8080 make test-integration

Documentation

Overview

Package sink provides a typed, concurrency-safe Go client for the Sink gRPC service.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DeleteResultsError

func DeleteResultsError(results []DeleteResult) error

func ReadResultsError

func ReadResultsError(results []ReadResult) error

func WriteResultsError

func WriteResultsError(results []WriteResult) error

Types

type Address

type Address struct {
	// contains filtered or unexported fields
}

Address identifies a record through logical names rather than physical database terminology.

func NewAddress

func NewAddress(store string, namespace string, dataset string, key Key) (Address, error)

func (Address) Dataset

func (a Address) Dataset() string

func (Address) Namespace

func (a Address) Namespace() string

func (Address) Store

func (a Address) Store() string

type BatchError

type BatchError struct {
	Failures []*OperationError
}

BatchError collects per-operation failures while preserving errors.Is and errors.As traversal through every failure.

func (*BatchError) Error

func (e *BatchError) Error() string

func (*BatchError) Unwrap

func (e *BatchError) Unwrap() []error

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is safe for concurrent use.

func Dial

func Dial(target string, opts DialOptions) (*Client, error)

Dial creates a lazily connected gRPC client. Call CheckHealth when startup must prove the endpoint is reachable before processing work.

func New

func New(connection grpc.ClientConnInterface, opts ClientOptions) (*Client, error)

New wraps an existing connection without taking ownership of it.

func (*Client) CheckHealth

func (c *Client) CheckHealth(ctx context.Context) error

CheckHealth verifies that the server's standard gRPC health service reports SERVING.

func (*Client) Close

func (c *Client) Close() error

Close closes connections created by Dial. It is a no-op for clients created with New because the caller owns that connection.

func (*Client) Delete

func (c *Client) Delete(
	ctx context.Context,
	completionMode CompletionMode,
	addresses ...Address,
) ([]DeleteResult, error)

Delete permanently deletes records. Deleting an absent record is successful. Large collections are split automatically, and transport failures are not retried. Earlier batches may have completed when a later batch returns an error.

func (*Client) Raw

func (c *Client) Raw() sinkv1.SinkClient

Raw returns the generated gRPC client for advanced or forward-compatible usage.

func (*Client) Read

func (c *Client) Read(ctx context.Context, addresses ...Address) ([]ReadResult, error)

Read returns results in request order and automatically splits large collections into configured operation-count batches. Transport-level Unavailable errors and retryable per-operation failures are retried because reads are idempotent. Returned operation indexes refer to the original collection.

func (*Client) Write

func (c *Client) Write(
	ctx context.Context,
	completionMode CompletionMode,
	operations ...WriteOperation,
) ([]WriteResult, error)

Write submits mixed put and merge operations and automatically splits large collections into configured operation-count batches. It deliberately does not retry transport failures because the server may already have applied or durably accepted the mutation. Earlier batches may have completed when a later batch returns an error.

type ClientOptions

type ClientOptions struct {
	MaxOperations          int
	ReadRetry              RetryPolicy
	MaxReceiveMessageBytes int
	MaxSendMessageBytes    int
}

ClientOptions controls request limits and safe read retries.

type Dataset added in v0.4.0

type Dataset struct {
	// contains filtered or unexported fields
}

Dataset provides validated, batch-native reads and mutations for one routing and encoding scope. Its methods return decoded results and a BatchError when any individual operation fails.

func NewDataset added in v0.4.0

func NewDataset(client *Client, opts DatasetOptions) (*Dataset, error)

NewDataset binds stable dataset settings to a Client. MergeProgram is optional and is copied when configured.

func (*Dataset) Create added in v0.4.0

func (d *Dataset) Create(
	ctx context.Context,
	completionMode CompletionMode,
	records ...Record,
) ([]WriteResult, error)

Create writes complete documents only when their keys do not already exist.

func (*Dataset) Merge added in v0.4.0

func (d *Dataset) Merge(
	ctx context.Context,
	completionMode CompletionMode,
	missingDocumentMode MissingDocumentMode,
	records ...Record,
) ([]WriteResult, error)

Merge atomically applies the Dataset's bound Lua program to every incoming record. A Dataset without MergeProgram rejects Merge before sending an RPC.

func (*Dataset) Read added in v0.4.0

func (d *Dataset) Read(ctx context.Context, keys ...Key) ([]ReadResult, error)

Read fetches one or more records by key. It preserves key order, splits large collections automatically, and treats not-found results as successful reads.

func (*Dataset) Replace added in v0.4.0

func (d *Dataset) Replace(
	ctx context.Context,
	completionMode CompletionMode,
	records ...Record,
) ([]WriteResult, error)

Replace writes complete documents only when their keys already exist.

func (*Dataset) Upsert added in v0.4.0

func (d *Dataset) Upsert(
	ctx context.Context,
	completionMode CompletionMode,
	records ...Record,
) ([]WriteResult, error)

Upsert writes complete documents whether or not their keys already exist.

type DatasetOptions added in v0.4.0

type DatasetOptions struct {
	Store        string
	Namespace    string
	Dataset      string
	Encoding     DocumentEncoding
	MergeProgram *LuaProgram
}

DatasetOptions binds the stable routing, encoding, and optional merge program shared by records in one logical dataset. Completion mode remains a per-call choice because callers of the same dataset can require different durability and visibility guarantees.

type DeleteResult

type DeleteResult struct {
	OperationIndex int
	Status         DeleteStatus
	Failure        *OperationError
}

func (DeleteResult) Err

func (r DeleteResult) Err() error

type DialOptions

type DialOptions struct {
	Client               ClientOptions
	TransportCredentials credentials.TransportCredentials
	GRPCOptions          []grpc.DialOption
}

DialOptions controls connection creation. TLS with a minimum version of 1.2 is used when TransportCredentials is nil. Use insecure.NewCredentials only for a trusted plaintext development endpoint.

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document contains one immutable, explicitly encoded user object.

func NewDocument

func NewDocument(value any, encoding DocumentEncoding) (Document, error)

NewDocument encodes a Go value using exactly the requested format. JSON uses json struct tags and BSON uses bson struct tags.

func NewRawDocument added in v0.4.0

func NewRawDocument(encoding DocumentEncoding, payload []byte) (Document, error)

NewRawDocument validates and copies an already encoded document payload.

func (Document) Decode added in v0.2.0

func (d Document) Decode(destination any) error

func (Document) Encoding added in v0.4.0

func (d Document) Encoding() DocumentEncoding

func (Document) Payload added in v0.4.0

func (d Document) Payload() []byte

type DocumentEncoding added in v0.4.0

type DocumentEncoding = sinkv1.DocumentEncoding

type Key

type Key struct {
	// contains filtered or unexported fields
}

Key is an immutable logical record key. Use one of the key constructors.

func BytesKey

func BytesKey(value []byte) Key

func Int64Key

func Int64Key(value int64) Key

func OpaqueKey

func OpaqueKey(typeName string, value []byte) (Key, error)

func StringKey

func StringKey(value string) Key

type LuaProgram added in v0.2.0

type LuaProgram struct {
	// contains filtered or unexported fields
}

LuaProgram is an immutable, self-contained merge rule. Sink verifies the digest and caches the compiled program while executing each merge in a fresh VM.

func NewLuaProgram added in v0.2.0

func NewLuaProgram(source []byte) (LuaProgram, error)

func (LuaProgram) SHA256 added in v0.2.0

func (p LuaProgram) SHA256() []byte

func (LuaProgram) Source added in v0.2.0

func (p LuaProgram) Source() []byte

type MergeOptions

type MergeOptions struct {
	Incoming            Document
	Program             LuaProgram
	MissingDocumentMode MissingDocumentMode
}

MergeOptions describes a Lua-driven read-modify-write operation.

type OperationError

type OperationError struct {
	OperationIndex int
	Code           FailureCode
	Message        string
	Retryable      bool
}

OperationError reports a per-record failure without discarding successful results from the same batch.

func (*OperationError) Error

func (e *OperationError) Error() string

type ProtocolError

type ProtocolError struct {
	Method  string
	Message string
}

ProtocolError means the server returned a structurally invalid response.

func (*ProtocolError) Error

func (e *ProtocolError) Error() string

type ReadResult

type ReadResult struct {
	OperationIndex int
	Status         ReadStatus
	Document       Document
	Revision       RevisionToken
	Failure        *OperationError
}

func (ReadResult) Err

func (r ReadResult) Err() error

type Record added in v0.4.0

type Record struct {
	Key   Key
	Value any
}

Record pairs one logical key with the Go value to encode for a Dataset mutation.

type RetryPolicy

type RetryPolicy struct {
	MaxAttempts    int
	InitialBackoff time.Duration
	MaxBackoff     time.Duration
	Multiplier     float64
	Jitter         float64
}

RetryPolicy controls retries for transport-level Unavailable errors and retryable per-operation failures from Read. Mutating RPCs are never retried automatically.

type RevisionToken

type RevisionToken struct {
	// contains filtered or unexported fields
}

RevisionToken is an opaque storage revision returned by Sink.

func (RevisionToken) Bytes

func (r RevisionToken) Bytes() []byte

type WriteOperation

type WriteOperation struct {
	// contains filtered or unexported fields
}

WriteOperation is either a put or merge operation. Use NewPut or NewMerge.

func NewMerge

func NewMerge(address Address, opts MergeOptions) (WriteOperation, error)

func NewPut

func NewPut(address Address, document Document, mode WriteMode) (WriteOperation, error)

type WriteResult

type WriteResult struct {
	OperationIndex int
	Status         WriteStatus
	Revision       RevisionToken
	Failure        *OperationError
}

func (WriteResult) Err

func (r WriteResult) Err() error

Directories

Path Synopsis
api
sink/v1
Package sinkv1 contains generated protobuf definitions for the Sink v1 gRPC service.
Package sinkv1 contains generated protobuf definitions for the Sink v1 gRPC service.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL