kinesis-consumer-go

README

Kinesis Consumer Go

A pure Go library for consuming Kinesis streams with shard leasing, shard-aware checkpointing, and reshard-aware ordering. No Java, no MultiLangDaemon.

Why this library

• Native Go, single process. No JVM sidecar or MultiLangDaemon.
• Modular packaging: import only the core, plus an optional backend module.
• Not tied to DynamoDB: coordination state lives behind pluggable checkpoint-store and lease-manager interfaces. A Valkey backend is built in; DynamoDB and Redis are planned.
• Reshard-aware ordering via SHARD_END markers and parent gating.
• Tunable batching, retries, polling, and per-shard concurrency.
• Production-safe fail-fast poison-record handling by default, with explicit skip or send-to-DLQ options.

Key features

• Shard leasing with heartbeats and fair-share rebalancing.
• Shard-aware checkpointing and reshard gating.
• Record or batch handlers with configurable batching and retry.
• Pluggable, bounded/retrying DLQ publisher interface with stable poison-record idempotency keys for downstream deduplication.
• Optional graceful drain mode on shutdown (finish in-flight work, checkpoint, release lease).
• Pluggable checkpoint stores and lease managers.
• Opt-in structured logging via log/slog (silent by default).
• Opt-in metrics with a dependency-free UDP statsd reporter and packaged Telegraf, InfluxDB, and Grafana assets (silent by default).
• LocalStack + Valkey workflow for local testing.

Install

go get github.com/ajaysinghpanwar2002/kinesis-consumer-go

Documentation

• Getting started — install, a minimal copy-paste-runnable consumer, multi-worker coordination, and graceful shutdown.
• Features and capabilities — a complete, source-accurate inventory of what the library does (and what it does not yet do).
• Configuration reference — every Config field and With* option with defaults, effects, and validation rules.
• Handler failure policy, DLQ, and shard concurrency
• Logging — enabling WithLogger, the complete structured event catalog with levels and attributes, and production guidance.
• Metrics — enabling WithMetrics, the complete metric and tag catalog, statsd wire conventions, and the Telegraf/InfluxDB/Grafana path.
• Integration test suite — a verifiable ledger of every integration scenario and the behavior it proves.

Examples

• examples/valkey — a runnable consumer backed by Valkey for both checkpoints and leasing. It passes the Valkey store to consumer.New and does not call WithLeaseManager: because the store implements lease.Provider, the consumer turns on shard leasing automatically. Run two copies against the same stream and Valkey to see shards spread across workers. It needs a real (or LocalStack) Kinesis endpoint and a reachable Valkey server, so it is its own module and is not part of the test suite:

  cd examples/valkey
  go run . -stream-name my-stream -consumer-group my-app -valkey-addr localhost:6379
  

Development

The repository is a Go workspace with four modules: the core library, the Valkey backend under pkg/backend/valkey, the example under examples/valkey, and the integration suite under test/integration. Because go test ./... does not cross module boundaries, the unit make targets iterate the first three so nothing is silently skipped; the infra-backed test/integration suite runs separately via make integration.

Run the unit test suite across all modules:

make test

Compile all packages (including the example):

make build

Other targets: make vet, make fmt-check, make tidy.

Run the core module's unit tests in Docker (the image runs go test ./... from the repository root, which covers the core library module only — the Valkey backend and example modules are exercised by make test and CI):

make docker-test

Git hooks

Shared hooks live in .githooks/ and are installed by pointing git at them:

make hooks   # sets core.hooksPath=.githooks

• pre-commit runs the full gate: make fmt-check, make vet, make build, make test.
• pre-push re-runs make test as a final safety net (e.g. for commits made with --no-verify).

Unit tests are hermetic (they use an in-memory Redis and no network).

Integration test suite

Beyond the hermetic unit tests, the library is covered by an integration suite that runs against real LocalStack Kinesis + real Valkey (not mocks) via make integration. It exercises 30+ scenarios across attribution, checkpoint resume, uncooperative failover, failure policies and DLQ, batch and concurrent processing, start positions, resharding, rebalance fairness, and key isolation.

Each hardening scenario carries a load-bearing assertion, was mutation-verified (a deliberate bug was shown to make it fail, then reverted), and was run repeatedly to rule out flakiness. See docs/testing.md for the full ledger — every row maps to a real test you can read and run.

Comparison: AWS KCL vs MultiLangDaemon vs this library

Aspect	AWS KCL (Java)	KCL MultiLangDaemon	Kinesis Consumer Go
Runtime	Java	App + Java daemon	Go only
Process model	Single JVM	Multi-process	Single process
Coordination store	DynamoDB (default)	DynamoDB (default)	Valkey built-in; pluggable interface (DynamoDB/Redis planned)
Packaging	Java deps	Java jars + wrapper	Core + optional modules
Language support	Java	Any via daemon	Go
Read path	Polling + enhanced fan-out	Polling + enhanced fan-out	Polling GetRecords only (shared 2 MB/s per shard; no SubscribeToShard)
KPL deaggregation	Built in	Built in (via daemon)	Not supported — raw records only; KPL-aggregated streams need a non-aggregating producer or handler-side deaggregation (see docs/features.md)
Delivery / shard exclusivity	At-least-once; brief dual-processing on lease handoff	At-least-once; brief dual-processing on lease handoff	At-least-once; bounded dual-processing windows on rebalance/failover (see docs/features.md)

Community Go alternatives exist, but most are either lightweight consumers or wrappers around the Java KCL. This project targets a native Go experience with KCL-like coordination, without the daemon.

Community native Go alternatives (non-AWS)

Library	Focus	Notes
vmware-go-kcl-v2	KCL-like API	Native Go; aims to mirror KCL interface.
gokini	Minimal deps	Explicitly no MultiLangDaemon.
kinsumer	Native consumer	Smaller API surface, not full KCL parity.
kinesumer	Consumer group client	Uses DynamoDB state store.
kinesis-consumer	Lightweight wrapper	Pluggable checkpoint backends.

These are community-maintained projects; scope and activity vary.

License

This project is licensed under the MIT License. See LICENSE.