sink

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: MIT Imports: 21 Imported by: 0

README

sink-go

sink-go is the typed, concurrency-safe Go client for the liran/sink gRPC service. It covers the full batch API: reads, puts, self-contained Lua merges, and hard deletes with synchronous or durable asynchronous completion.

Install

go get github.com/liran/sink-go

The client requires Go 1.27 or newer.

Quick start

Dial uses TLS 1.2 or newer by default. The explicit insecure credentials in this example are appropriate only for a trusted local endpoint.

package main

import (
	"context"
	"log"

	sink "github.com/liran/sink-go"
	"google.golang.org/grpc/credentials/insecure"
)

func main() {
	dialOptions := sink.DialOptions{
		TransportCredentials: insecure.NewCredentials(),
	}
	client, err := sink.Dial("127.0.0.1:8080", dialOptions)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	address, err := sink.NewAddress(
		"primary",
		"catalog",
		"products",
		sink.StringKey("product-42"),
	)
	if err != nil {
		log.Fatal(err)
	}
	value := map[string]any{"name": "keyboard", "stock": 12}
	operation, err := sink.NewPut(address, value, sink.WriteUpsert)
	if err != nil {
		log.Fatal(err)
	}

	results, err := client.Write(
		context.Background(),
		sink.CompletionWaitUntilVisible,
		operation,
	)
	if err != nil {
		log.Fatal(err)
	}
	if results[0].Failure != nil {
		log.Fatal(results[0].Failure)
	}
}

NewPut and NewMerge accept ordinary Go values and encode them as JSON inside the client. The same application code works with MongoDB, Elasticsearch, and OpenSearch; storage-specific encoding stays inside the server adapter. Reads return a Document whose Decode method unmarshals into a caller-provided Go value and whose JSON method returns an immutable copy of the encoded object. time.Time values are encoded as ordinary RFC3339 JSON strings. The client also records their JSON Pointer paths so MongoDB can store them as BSON datetimes; Lua and search storage still receive strings, and unrelated RFC3339-looking strings are not retyped.

A merge rule is ordinary application source code. Construct it once and reuse the immutable LuaProgram across operations; the client includes its SHA-256 digest automatically:

source := []byte(`
return function(current, incoming)
    current = current or json.object()
    current.stock = incoming.stock
    current.updated_at = sink.v1.time.now()
    return current
end`)
program, err := sink.NewLuaProgram(source)
if err != nil {
    log.Fatal(err)
}
mergeOptions := sink.MergeOptions{
    Incoming:            map[string]any{"stock": 12},
    Program:             program,
    MissingDocumentMode: sink.MissingDocumentCreate,
}
operation, err := sink.NewMerge(address, mergeOptions)
if err != nil {
    log.Fatal(err)
}

The merge function receives only current and incoming. Sink provides versioned sink.v1 array, object, and retry-stable time helpers. See the Lua merge developer guide for the complete function reference and reliability rules.

API model

  • Read(ctx, addresses...) preserves request order and reports found, not-found, or failed results independently. ReadAll automatically splits a larger collection by the configured operation limit.
  • Write(ctx, completionMode, operations...) supports mixed put and merge batches. Use NewPut and NewMerge to construct validated operations; WriteAll handles larger collections.
  • Delete(ctx, completionMode, addresses...) performs hard deletes; deleting an absent record is successful. DeleteAll handles larger collections.
  • String, int64, byte, and opaque legacy keys are supported.
  • CheckHealth uses the standard gRPC health service.
  • Raw exposes the generated api/sink/v1 client for advanced use.

Use CompletionWaitUntilApplied for storage acknowledgement, CompletionWaitUntilVisible when a following search read must observe the mutation, and CompletionReturnAfterAccepted for durable asynchronous queueing. For OpenSearch and Elasticsearch, visible completion maps to refresh=wait_for; MongoDB is immediately visible after an acknowledged write.

Lua source travels with the merge intent in both synchronous and asynchronous mode. Sink caches compilation by digest while using a fresh VM per execution, so rules follow application releases without server-side profile management. The client declares identical source only once per Write batch; Sink embeds the full program into each asynchronous Kafka mutation so it remains replayable without server-side rule state.

Each result includes its operation index. The client validates result counts, indexes, statuses, documents, and failure details before returning a response. Per-record failures are represented by OperationError, so one bad record does not hide successful records from the same batch. Each result exposes Err(), and ReadResultsError, WriteResultsError, and DeleteResultsError collect all operation failures into an errors.As-compatible BatchError.

Reliability behavior

Reads retry transport-level Unavailable failures and retryable per-operation failures with bounded exponential backoff and jitter. Only failed operations are resubmitted after a partial batch response. The default is three attempts, starting at 100 ms and capped at one second; ClientOptions.ReadRetry can tune or disable retries by setting MaxAttempts to one.

Writes and deletes are never retried automatically. A transport error can arrive after Sink has already applied a synchronous mutation or durably accepted an asynchronous one, so automatic mutation retries could duplicate work. Callers should retry only when their operation is safe under Sink's documented at-least-once semantics.

The client limits batches to 1,000 operations by default, matching Sink's default configuration. Set ClientOptions.MaxOperations when the server is configured with a different limit. Encoded requests and responses are limited to 64 MiB by default; use MaxSendMessageBytes and MaxReceiveMessageBytes to match custom server limits. A Client and its underlying gRPC connection are safe for concurrent use.

Compatibility and development

The generated protocol matches the current Sink server contract. CI runs descriptor contract tests, race-enabled unit tests against an in-memory gRPC server, malformed-response tests, static analysis, and an end-to-end compatibility test against the current Sink main branch with MongoDB and Kafka. The compatibility workflow also runs weekly so server-side drift is detected without requiring a client commit.

make test
make test-race
make lint
make proto-check

To run the external compatibility test against an already running server:

SINK_INTEGRATION_ADDRESS=127.0.0.1:8080 make test-integration

Documentation

Overview

Package sink provides a typed, concurrency-safe Go client for the Sink gRPC service.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DeleteResultsError

func DeleteResultsError(results []DeleteResult) error

func ReadResultsError

func ReadResultsError(results []ReadResult) error

func WriteResultsError

func WriteResultsError(results []WriteResult) error

Types

type Address

type Address struct {
	// contains filtered or unexported fields
}

Address identifies a record through logical names rather than physical database terminology.

func NewAddress

func NewAddress(store string, namespace string, dataset string, key Key) (Address, error)

func (Address) Dataset

func (a Address) Dataset() string

func (Address) Namespace

func (a Address) Namespace() string

func (Address) Store

func (a Address) Store() string

type BatchError

type BatchError struct {
	Failures []*OperationError
}

BatchError collects per-operation failures while preserving errors.Is and errors.As traversal through every failure.

func (*BatchError) Error

func (e *BatchError) Error() string

func (*BatchError) Unwrap

func (e *BatchError) Unwrap() []error

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is safe for concurrent use.

func Dial

func Dial(target string, opts DialOptions) (*Client, error)

Dial creates a lazily connected gRPC client. Call CheckHealth when startup must prove the endpoint is reachable before processing work.

func New

func New(connection grpc.ClientConnInterface, opts ClientOptions) (*Client, error)

New wraps an existing connection without taking ownership of it.

func (*Client) CheckHealth

func (c *Client) CheckHealth(ctx context.Context) error

CheckHealth verifies that the server's standard gRPC health service reports SERVING.

func (*Client) Close

func (c *Client) Close() error

Close closes connections created by Dial. It is a no-op for clients created with New because the caller owns that connection.

func (*Client) Delete

func (c *Client) Delete(
	ctx context.Context,
	completionMode CompletionMode,
	addresses ...Address,
) ([]DeleteResult, error)

Delete permanently deletes records. Deleting an absent record is successful. Transport failures are not retried automatically.

func (*Client) DeleteAll

func (c *Client) DeleteAll(
	ctx context.Context,
	completionMode CompletionMode,
	addresses []Address,
) ([]DeleteResult, error)

DeleteAll splits a larger collection into configured operation-count batches. Earlier batches may have completed when a later batch returns an RPC error.

func (*Client) Raw

func (c *Client) Raw() sinkv1.SinkClient

Raw returns the generated gRPC client for advanced or forward-compatible usage.

func (*Client) Read

func (c *Client) Read(ctx context.Context, addresses ...Address) ([]ReadResult, error)

Read returns results in request order. Transport-level Unavailable errors and retryable per-operation failures are retried, because reads are idempotent.

func (*Client) ReadAll

func (c *Client) ReadAll(ctx context.Context, addresses []Address) ([]ReadResult, error)

ReadAll splits a larger collection into configured operation-count batches. Returned operation indexes refer to the original collection.

func (*Client) Write

func (c *Client) Write(
	ctx context.Context,
	completionMode CompletionMode,
	operations ...WriteOperation,
) ([]WriteResult, error)

Write submits a mixed batch of put and merge operations. It deliberately does not retry transport failures because the server may already have applied or durably accepted the mutation.

func (*Client) WriteAll

func (c *Client) WriteAll(
	ctx context.Context,
	completionMode CompletionMode,
	operations []WriteOperation,
) ([]WriteResult, error)

WriteAll splits a larger collection into configured operation-count batches. Earlier batches may have completed when a later batch returns an RPC error.

type ClientOptions

type ClientOptions struct {
	MaxOperations          int
	ReadRetry              RetryPolicy
	MaxReceiveMessageBytes int
	MaxSendMessageBytes    int
}

ClientOptions controls request limits and safe read retries.

type DeleteResult

type DeleteResult struct {
	OperationIndex int
	Status         DeleteStatus
	Failure        *OperationError
}

func (DeleteResult) Err

func (r DeleteResult) Err() error

type DialOptions

type DialOptions struct {
	Client               ClientOptions
	TransportCredentials credentials.TransportCredentials
	GRPCOptions          []grpc.DialOption
}

DialOptions controls connection creation. TLS with a minimum version of 1.2 is used when TransportCredentials is nil. Use insecure.NewCredentials only for a trusted plaintext development endpoint.

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document contains an immutable JSON object returned by Sink.

func (Document) Decode added in v0.2.0

func (d Document) Decode(destination any) error

func (Document) JSON added in v0.2.0

func (d Document) JSON() []byte

type Key

type Key struct {
	// contains filtered or unexported fields
}

Key is an immutable logical record key. Use one of the key constructors.

func BytesKey

func BytesKey(value []byte) Key

func Int64Key

func Int64Key(value int64) Key

func OpaqueKey

func OpaqueKey(typeName string, value []byte) (Key, error)

func StringKey

func StringKey(value string) Key

type LuaProgram added in v0.2.0

type LuaProgram struct {
	// contains filtered or unexported fields
}

LuaProgram is an immutable, self-contained merge rule. Sink verifies the digest and caches the compiled program while executing each merge in a fresh VM.

func NewLuaProgram added in v0.2.0

func NewLuaProgram(source []byte) (LuaProgram, error)

func (LuaProgram) SHA256 added in v0.2.0

func (p LuaProgram) SHA256() []byte

func (LuaProgram) Source added in v0.2.0

func (p LuaProgram) Source() []byte

type MergeOptions

type MergeOptions struct {
	Incoming            any
	Program             LuaProgram
	MissingDocumentMode MissingDocumentMode
}

MergeOptions describes a Lua-driven read-modify-write operation.

type OperationError

type OperationError struct {
	OperationIndex int
	Code           FailureCode
	Message        string
	Retryable      bool
}

OperationError reports a per-record failure without discarding successful results from the same batch.

func (*OperationError) Error

func (e *OperationError) Error() string

type ProtocolError

type ProtocolError struct {
	Method  string
	Message string
}

ProtocolError means the server returned a structurally invalid response.

func (*ProtocolError) Error

func (e *ProtocolError) Error() string

type ReadResult

type ReadResult struct {
	OperationIndex int
	Status         ReadStatus
	Document       Document
	Revision       RevisionToken
	Failure        *OperationError
}

func (ReadResult) Err

func (r ReadResult) Err() error

type RetryPolicy

type RetryPolicy struct {
	MaxAttempts    int
	InitialBackoff time.Duration
	MaxBackoff     time.Duration
	Multiplier     float64
	Jitter         float64
}

RetryPolicy controls retries for transport-level Unavailable errors and retryable per-operation failures from Read. Mutating RPCs are never retried automatically.

type RevisionToken

type RevisionToken struct {
	// contains filtered or unexported fields
}

RevisionToken is an opaque storage revision returned by Sink.

func (RevisionToken) Bytes

func (r RevisionToken) Bytes() []byte

type WriteOperation

type WriteOperation struct {
	// contains filtered or unexported fields
}

WriteOperation is either a put or merge operation. Use NewPut or NewMerge.

func NewMerge

func NewMerge(address Address, opts MergeOptions) (WriteOperation, error)

func NewPut

func NewPut(address Address, value any, mode WriteMode) (WriteOperation, error)

type WriteResult

type WriteResult struct {
	OperationIndex int
	Status         WriteStatus
	Revision       RevisionToken
	Failure        *OperationError
}

func (WriteResult) Err

func (r WriteResult) Err() error

Directories

Path Synopsis
api
sink/v1
Package sinkv1 contains generated protobuf definitions for the Sink v1 gRPC service.
Package sinkv1 contains generated protobuf definitions for the Sink v1 gRPC service.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL