sink

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 28, 2026 License: MIT Imports: 15 Imported by: 0

README

sink-go

sink-go is the typed, concurrency-safe Go client for the liran/sink gRPC service. It covers the full batch API: reads, puts, profile-driven merges, and hard deletes with synchronous or durable asynchronous completion.

Install

go get github.com/liran/sink-go

The client requires Go 1.27 or newer.

Quick start

Dial uses TLS 1.2 or newer by default. The explicit insecure credentials in this example are appropriate only for a trusted local endpoint.

package main

import (
	"context"
	"log"

	sink "github.com/liran/sink-go"
	"google.golang.org/grpc/credentials/insecure"
)

func main() {
	dialOptions := sink.DialOptions{
		TransportCredentials: insecure.NewCredentials(),
	}
	client, err := sink.Dial("127.0.0.1:8080", dialOptions)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	address, err := sink.NewAddress(
		"primary",
		"catalog",
		"products",
		sink.StringKey("product-42"),
	)
	if err != nil {
		log.Fatal(err)
	}
	value := map[string]any{"name": "keyboard", "stock": 12}
	document, err := sink.JSONDocument(value)
	if err != nil {
		log.Fatal(err)
	}
	operation, err := sink.NewPut(address, document, sink.WriteUpsert)
	if err != nil {
		log.Fatal(err)
	}

	results, err := client.Write(
		context.Background(),
		sink.CompletionWaitUntilApplied,
		operation,
	)
	if err != nil {
		log.Fatal(err)
	}
	if results[0].Failure != nil {
		log.Fatal(results[0].Failure)
	}
}

Use application/bson documents with MongoDB-backed Sink deployments and application/json with Elasticsearch or OpenSearch. NewDocument accepts already encoded bytes without adding a storage-specific dependency to this module.

API model

  • Read(ctx, addresses...) preserves request order and reports found, not-found, or failed results independently. ReadAll automatically splits a larger collection by the configured operation limit.
  • Write(ctx, completionMode, operations...) supports mixed put and merge batches. Use NewPut and NewMerge to construct validated operations; WriteAll handles larger collections.
  • Delete(ctx, completionMode, addresses...) performs hard deletes; deleting an absent record is successful. DeleteAll handles larger collections.
  • String, int64, byte, and opaque legacy keys are supported.
  • CheckHealth uses the standard gRPC health service.
  • Raw exposes the generated api/sink/v1 client for advanced use.

The stock Sink server registers the json-merge-patch@1 merge profile for JSON objects. Deployments can add application-specific versioned profiles.

Each result includes its operation index. The client validates result counts, indexes, statuses, documents, and failure details before returning a response. Per-record failures are represented by OperationError, so one bad record does not hide successful records from the same batch. Each result exposes Err(), and ReadResultsError, WriteResultsError, and DeleteResultsError collect all operation failures into an errors.As-compatible BatchError.

Reliability behavior

Reads retry transport-level Unavailable failures and retryable per-operation failures with bounded exponential backoff and jitter. Only failed operations are resubmitted after a partial batch response. The default is three attempts, starting at 100 ms and capped at one second; ClientOptions.ReadRetry can tune or disable retries by setting MaxAttempts to one.

Writes and deletes are never retried automatically. A transport error can arrive after Sink has already applied a synchronous mutation or durably accepted an asynchronous one, so automatic mutation retries could duplicate work. Callers should retry only when their operation is safe under Sink's documented at-least-once semantics.

The client limits batches to 1,000 operations by default, matching Sink v0.1.0. Set ClientOptions.MaxOperations when the server is configured with a different limit. Encoded requests and responses are limited to 64 MiB by default; use MaxSendMessageBytes and MaxReceiveMessageBytes to match custom server limits. A Client and its underlying gRPC connection are safe for concurrent use.

Compatibility and development

The generated protocol matches Sink v0.1.0 and the current server contract. CI runs descriptor contract tests, race-enabled unit tests against an in-memory gRPC server, malformed-response tests, static analysis, and an end-to-end compatibility test against the current Sink main branch with MongoDB and Kafka. The compatibility workflow also runs weekly so server-side drift is detected without requiring a client commit.

make test
make test-race
make lint
make proto-check

To run the external compatibility test against an already running server:

SINK_INTEGRATION_ADDRESS=127.0.0.1:8080 make test-integration

Documentation

Overview

Package sink provides a typed, concurrency-safe Go client for the Sink gRPC service.

Index

Constants

View Source
const (
	ContentTypeBSON = "application/bson"
	ContentTypeJSON = "application/json"
)

Variables

This section is empty.

Functions

func DeleteResultsError

func DeleteResultsError(results []DeleteResult) error

func ReadResultsError

func ReadResultsError(results []ReadResult) error

func WriteResultsError

func WriteResultsError(results []WriteResult) error

Types

type Address

type Address struct {
	// contains filtered or unexported fields
}

Address identifies a record through logical names rather than physical database terminology.

func NewAddress

func NewAddress(store string, namespace string, dataset string, key Key) (Address, error)

func (Address) Dataset

func (a Address) Dataset() string

func (Address) Namespace

func (a Address) Namespace() string

func (Address) Store

func (a Address) Store() string

type BatchError

type BatchError struct {
	Failures []*OperationError
}

BatchError collects per-operation failures while preserving errors.Is and errors.As traversal through every failure.

func (*BatchError) Error

func (e *BatchError) Error() string

func (*BatchError) Unwrap

func (e *BatchError) Unwrap() []error

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is safe for concurrent use.

func Dial

func Dial(target string, opts DialOptions) (*Client, error)

Dial creates a lazily connected gRPC client. Call CheckHealth when startup must prove the endpoint is reachable before processing work.

func New

func New(connection grpc.ClientConnInterface, opts ClientOptions) (*Client, error)

New wraps an existing connection without taking ownership of it.

func (*Client) CheckHealth

func (c *Client) CheckHealth(ctx context.Context) error

CheckHealth verifies that the server's standard gRPC health service reports SERVING.

func (*Client) Close

func (c *Client) Close() error

Close closes connections created by Dial. It is a no-op for clients created with New because the caller owns that connection.

func (*Client) Delete

func (c *Client) Delete(
	ctx context.Context,
	completionMode CompletionMode,
	addresses ...Address,
) ([]DeleteResult, error)

Delete permanently deletes records. Deleting an absent record is successful. Transport failures are not retried automatically.

func (*Client) DeleteAll

func (c *Client) DeleteAll(
	ctx context.Context,
	completionMode CompletionMode,
	addresses []Address,
) ([]DeleteResult, error)

DeleteAll splits a larger collection into configured operation-count batches. Earlier batches may have completed when a later batch returns an RPC error.

func (*Client) Raw

func (c *Client) Raw() sinkv1.SinkClient

Raw returns the generated gRPC client for advanced or forward-compatible usage.

func (*Client) Read

func (c *Client) Read(ctx context.Context, addresses ...Address) ([]ReadResult, error)

Read returns results in request order. Transport-level Unavailable errors and retryable per-operation failures are retried, because reads are idempotent.

func (*Client) ReadAll

func (c *Client) ReadAll(ctx context.Context, addresses []Address) ([]ReadResult, error)

ReadAll splits a larger collection into configured operation-count batches. Returned operation indexes refer to the original collection.

func (*Client) Write

func (c *Client) Write(
	ctx context.Context,
	completionMode CompletionMode,
	operations ...WriteOperation,
) ([]WriteResult, error)

Write submits a mixed batch of put and merge operations. It deliberately does not retry transport failures because the server may already have applied or durably accepted the mutation.

func (*Client) WriteAll

func (c *Client) WriteAll(
	ctx context.Context,
	completionMode CompletionMode,
	operations []WriteOperation,
) ([]WriteResult, error)

WriteAll splits a larger collection into configured operation-count batches. Earlier batches may have completed when a later batch returns an RPC error.

type ClientOptions

type ClientOptions struct {
	MaxOperations          int
	ReadRetry              RetryPolicy
	MaxReceiveMessageBytes int
	MaxSendMessageBytes    int
}

ClientOptions controls request limits and safe read retries.

type DeleteResult

type DeleteResult struct {
	OperationIndex int
	Status         DeleteStatus
	Failure        *OperationError
}

func (DeleteResult) Err

func (r DeleteResult) Err() error

type DialOptions

type DialOptions struct {
	Client               ClientOptions
	TransportCredentials credentials.TransportCredentials
	GRPCOptions          []grpc.DialOption
}

DialOptions controls connection creation. TLS with a minimum version of 1.2 is used when TransportCredentials is nil. Use insecure.NewCredentials only for a trusted plaintext development endpoint.

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document contains content-type-labelled, lossless bytes. NewDocument copies data so later caller mutation cannot change an in-flight operation.

func JSONDocument

func JSONDocument(value any) (Document, error)

func NewDocument

func NewDocument(contentType string, data []byte) (Document, error)

func (Document) Bytes

func (d Document) Bytes() []byte

func (Document) ContentType

func (d Document) ContentType() string

func (Document) DecodeJSON

func (d Document) DecodeJSON(destination any) error

type Key

type Key struct {
	// contains filtered or unexported fields
}

Key is an immutable logical record key. Use one of the key constructors.

func BytesKey

func BytesKey(value []byte) Key

func Int64Key

func Int64Key(value int64) Key

func OpaqueKey

func OpaqueKey(typeName string, value []byte) (Key, error)

func StringKey

func StringKey(value string) Key

type MergeOptions

type MergeOptions struct {
	IncomingDocument    Document
	Profile             MergeProfile
	MissingDocumentMode MissingDocumentMode
}

MergeOptions describes a profile-driven read-modify-write operation.

type MergeProfile

type MergeProfile struct {
	// contains filtered or unexported fields
}

MergeProfile selects a server-side merge implementation and version.

func NewMergeProfile

func NewMergeProfile(name string, version uint64) (MergeProfile, error)

type OperationError

type OperationError struct {
	OperationIndex int
	Code           FailureCode
	Message        string
	Retryable      bool
}

OperationError reports a per-record failure without discarding successful results from the same batch.

func (*OperationError) Error

func (e *OperationError) Error() string

type ProtocolError

type ProtocolError struct {
	Method  string
	Message string
}

ProtocolError means the server returned a structurally invalid response.

func (*ProtocolError) Error

func (e *ProtocolError) Error() string

type ReadResult

type ReadResult struct {
	OperationIndex int
	Status         ReadStatus
	Document       Document
	Revision       RevisionToken
	Failure        *OperationError
}

func (ReadResult) Err

func (r ReadResult) Err() error

type RetryPolicy

type RetryPolicy struct {
	MaxAttempts    int
	InitialBackoff time.Duration
	MaxBackoff     time.Duration
	Multiplier     float64
	Jitter         float64
}

RetryPolicy controls retries for transport-level Unavailable errors and retryable per-operation failures from Read. Mutating RPCs are never retried automatically.

type RevisionToken

type RevisionToken struct {
	// contains filtered or unexported fields
}

RevisionToken is an opaque storage revision returned by Sink.

func (RevisionToken) Bytes

func (r RevisionToken) Bytes() []byte

type WriteOperation

type WriteOperation struct {
	// contains filtered or unexported fields
}

WriteOperation is either a put or merge operation. Use NewPut or NewMerge.

func NewMerge

func NewMerge(address Address, opts MergeOptions) (WriteOperation, error)

func NewPut

func NewPut(address Address, document Document, mode WriteMode) (WriteOperation, error)

type WriteResult

type WriteResult struct {
	OperationIndex int
	Status         WriteStatus
	Revision       RevisionToken
	Failure        *OperationError
}

func (WriteResult) Err

func (r WriteResult) Err() error

Directories

Path Synopsis
api
sink/v1
Package sinkv1 contains generated protobuf definitions for the Sink v1 gRPC service.
Package sinkv1 contains generated protobuf definitions for the Sink v1 gRPC service.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL