syncinserter

README

syncinserter

syncinserter is a go package which provides abstraction to merge concurrency inserts into batch insert.

benchmarks

$ go clean -testcache && go test ./... && go test -timeout 5m -bench=. -benchtime=1000000x -benchmem ./...
ok      github.com/alexdyukov/syncinserter      4.538s
goos: linux
goarch: amd64
pkg: github.com/alexdyukov/syncinserter
cpu: AMD Ryzen 7 8845H w/ Radeon 780M Graphics
BenchmarkOverhead-16             1000000              6144 ns/op             156 B/op          1 allocs/op
BenchmarkPostgres-16             1000000             14973 ns/op             898 B/op         23 allocs/op
BenchmarkClickhouse-16           1000000             19956 ns/op             311 B/op          4 allocs/op
BenchmarkCassandra-16            1000000             33201 ns/op            1081 B/op         11 allocs/op
PASS
ok      github.com/alexdyukov/syncinserter      128.664s

License

MIT licensed. See the included LICENSE file for details.

Documentation

Overview

Package syncinserter provides batch inserter that merges concurrent insert operations into batch insert.

Index

• Variables
• type SyncInserter
  • func New(ctx context.Context, batchInsertFunc func(rows [][]any) error, ...) (*SyncInserter, error)
  • func (syncInserter *SyncInserter) Insert(ctx context.Context, row []any) error

Variables

var (
	// ErrInvalidMaxBatchSize indicates invalid max batch size parameter in New.
	ErrInvalidMaxBatchSize = errors.New("syncinserter: batch size should be greater 0")
	// ErrInvalidPeriod indicates invalid period parameter in New.
	ErrInvalidPeriod = errors.New("syncinserter: period should be greater 0")
)

Types

type SyncInserter

type SyncInserter struct {
	// contains filtered or unexported fields
}

SyncInserter is bulk inserter with sync Insert API. It merges all Insert calls during period and merges them in one batch call.

func New

func New(ctx context.Context, batchInsertFunc func(rows [][]any) error, maxBatchSize int, period time.Duration) (*SyncInserter, error)

New initializes SyncInserter.

func (*SyncInserter) Insert

func (syncInserter *SyncInserter) Insert(ctx context.Context, row []any) error

Insert adds a row to the current batch and blocks until context canceled or the batch is processed.