Documentation
¶
Overview ¶
Amazon kinesis producer A KPL-like batch producer for Amazon Kinesis built on top of the official Go AWS SDK and using the same aggregation format that KPL use.
Note: this project start as a fork of `tj/go-kinesis`. if you are not intersting in the KPL aggregation logic, you probably want to check it out.
Example ¶
logger := &StdLogger{log.New(os.Stdout, "", log.LstdFlags)}
cfg, _ := config.LoadDefaultConfig(context.TODO())
client := kinesis.NewFromConfig(cfg)
pr, err := New(&Config{
StreamName: "test",
BacklogCount: 2000,
Client: client,
Logger: logger,
})
if err != nil {
logger.Error("failed to create producer", err)
return
}
pr.Start()
// Handle failures
go func() {
for r := range pr.NotifyFailures() {
// r contains `Data`, `PartitionKey` and `Error()`
logger.Error("detected put failure", r.error)
}
}()
go func() {
for i := 0; i < 5000; i++ {
err := pr.Put([]byte("foo"), "bar")
if err != nil {
logger.Error("error producing", err)
}
}
}()
time.Sleep(3 * time.Second)
pr.Stop()
Index ¶
- Variables
- type AggregatedRecord
- func (*AggregatedRecord) Descriptor() ([]byte, []int)deprecated
- func (x *AggregatedRecord) GetExplicitHashKeyTable() []string
- func (x *AggregatedRecord) GetPartitionKeyTable() []string
- func (x *AggregatedRecord) GetRecords() []*Record
- func (*AggregatedRecord) ProtoMessage()
- func (x *AggregatedRecord) ProtoReflect() protoreflect.Message
- func (x *AggregatedRecord) Reset()
- func (x *AggregatedRecord) String() string
- type Aggregator
- func (a *Aggregator) Count() int
- func (a *Aggregator) Drain() ([]*ktypes.PutRecordsRequestEntry, error)
- func (a *Aggregator) DrainReady() ([]*ktypes.PutRecordsRequestEntry, error)
- func (a *Aggregator) GetShardCount() int
- func (a *Aggregator) HasRecords() bool
- func (a *Aggregator) Put(data []byte, partitionKey string)
- func (a *Aggregator) Size() int
- type AggregatorConfig
- type Config
- type FailureRecord
- type Limiter
- type LimiterConfig
- type LogValue
- type Logger
- type Producer
- type Putter
- type Record
- func (*Record) Descriptor() ([]byte, []int)deprecated
- func (x *Record) GetData() []byte
- func (x *Record) GetExplicitHashKeyIndex() uint64
- func (x *Record) GetPartitionKeyIndex() uint64
- func (x *Record) GetTags() []*Tag
- func (*Record) ProtoMessage()
- func (x *Record) ProtoReflect() protoreflect.Message
- func (x *Record) Reset()
- func (x *Record) String() string
- type ShardAggregator
- func (sa *ShardAggregator) Count() int
- func (sa *ShardAggregator) Drain() (*ktypes.PutRecordsRequestEntry, error)
- func (sa *ShardAggregator) IsEmpty() bool
- func (sa *ShardAggregator) Put(data []byte, partitionKey string) bool
- func (sa *ShardAggregator) ShardID() string
- func (sa *ShardAggregator) ShouldFlush() bool
- func (sa *ShardAggregator) Size() int
- type ShardLimiter
- type ShardMap
- type ShardMapState
- type ShardRange
- type StdLogger
- type Tag
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrStoppedProducer = errors.New("Unable to Put record. Producer is already stopped") ErrIllegalPartitionKey = errors.New("Invalid partition key. Length must be at least 1 and at most 256") ErrRecordSizeExceeded = errors.New("Data must be less than or equal to 1MB in size") )
Errors
var File_messages_proto protoreflect.FileDescriptor
Functions ¶
This section is empty.
Types ¶
type AggregatedRecord ¶
type AggregatedRecord struct {
PartitionKeyTable []string `protobuf:"bytes,1,rep,name=partition_key_table,json=partitionKeyTable" json:"partition_key_table,omitempty"`
ExplicitHashKeyTable []string `protobuf:"bytes,2,rep,name=explicit_hash_key_table,json=explicitHashKeyTable" json:"explicit_hash_key_table,omitempty"`
Records []*Record `protobuf:"bytes,3,rep,name=records" json:"records,omitempty"`
// contains filtered or unexported fields
}
func (*AggregatedRecord) Descriptor
deprecated
func (*AggregatedRecord) Descriptor() ([]byte, []int)
Deprecated: Use AggregatedRecord.ProtoReflect.Descriptor instead.
func (*AggregatedRecord) GetExplicitHashKeyTable ¶
func (x *AggregatedRecord) GetExplicitHashKeyTable() []string
func (*AggregatedRecord) GetPartitionKeyTable ¶
func (x *AggregatedRecord) GetPartitionKeyTable() []string
func (*AggregatedRecord) GetRecords ¶
func (x *AggregatedRecord) GetRecords() []*Record
func (*AggregatedRecord) ProtoMessage ¶
func (*AggregatedRecord) ProtoMessage()
func (*AggregatedRecord) ProtoReflect ¶
func (x *AggregatedRecord) ProtoReflect() protoreflect.Message
func (*AggregatedRecord) Reset ¶
func (x *AggregatedRecord) Reset()
func (*AggregatedRecord) String ¶
func (x *AggregatedRecord) String() string
type Aggregator ¶
type Aggregator struct {
// contains filtered or unexported fields
}
Aggregator coordinates multiple ShardAggregators for per-shard aggregation.
Routing Strategy:
Routes records by target shard ID (computed from partition key MD5 hash). ShardMap is required to determine the target shard for each partition key.
This matches Java KPL behavior where per-shard aggregation is always enabled and routes records based on shard boundaries.
Example:
shardMap, _ := NewShardMap(streamName, client, logger, false, 30*time.Second) aggregator := NewAggregator(shardMap, config) aggregator.Put(data, "random-uuid-123") // Routes to shard based on hash
func NewAggregator ¶
func NewAggregator(shardMap *ShardMap, config *AggregatorConfig) (*Aggregator, error)
NewAggregator creates a new Aggregator with a required ShardMap. ShardMap must not be nil as it is required for per-shard aggregation (matching Java KPL behavior). Returns an error if ShardMap is nil.
func (*Aggregator) Count ¶
func (a *Aggregator) Count() int
Count returns the total number of records across all shard aggregators.
func (*Aggregator) Drain ¶
func (a *Aggregator) Drain() ([]*ktypes.PutRecordsRequestEntry, error)
Drain returns all records from all aggregators, regardless of flush conditions. This is useful for flushing on shutdown or periodic flushes. After draining, empty aggregators are removed.
func (*Aggregator) DrainReady ¶
func (a *Aggregator) DrainReady() ([]*ktypes.PutRecordsRequestEntry, error)
DrainReady returns all aggregated records that should be flushed based on flush conditions. This is the recommended method for periodic flushes when you only want to drain records that have reached their flush thresholds, as opposed to Drain() which drains all records.
func (*Aggregator) GetShardCount ¶
func (a *Aggregator) GetShardCount() int
GetShardCount returns the number of distinct shards/keys being aggregated.
func (*Aggregator) HasRecords ¶
func (a *Aggregator) HasRecords() bool
HasRecords returns true if any aggregator has records.
func (*Aggregator) Put ¶
func (a *Aggregator) Put(data []byte, partitionKey string)
Put adds a record to the appropriate shard aggregator. Routes by shard ID (computed from partition key hash via ShardMap).
func (*Aggregator) Size ¶
func (a *Aggregator) Size() int
Size returns the total bytes across all shard aggregators.
type AggregatorConfig ¶
type AggregatorConfig struct {
MaxSize int // max bytes per aggregated record (default 50KB)
MaxCount int // max records per aggregated record (default 1000)
}
AggregatorConfig holds configuration for shard aggregators.
type Config ¶
type Config struct {
// StreamName is the Kinesis stream.
StreamName string
// FlushInterval is a regular interval for flushing the buffer. Defaults to 5s.
FlushInterval time.Duration
// BatchCount determine the maximum number of items to pack in batch.
// Must not exceed length. Defaults to 500.
BatchCount int
// BatchSize determine the maximum number of bytes to send with a PutRecords request.
// Must not exceed 5MiB; Default to 5MiB.
BatchSize int
// AggregateBatchCount determine the maximum number of items to pack into an aggregated record.
AggregateBatchCount int
// AggregationBatchSize determine the maximum number of bytes to pack into an aggregated record. User records larger
// than this will bypass aggregation.
AggregateBatchSize int
// BacklogCount determines the channel capacity before Put() will begin blocking. Default to `BatchCount`.
BacklogCount int
// Number of requests to sent concurrently. Default to 24.
MaxConnections int
// Logger is the logger used. Default to producer.Logger.
Logger Logger
// Enabling verbose logging. Default to false.
Verbose bool
// Client is the Putter interface implementation.
Client Putter
// RateLimit sets the maximum allowed put rate for a shard, as a percentage of backend limits.
// Like Java KPL, rate limiting is always enabled and cannot be disabled.
// The rate limit is implemented using a token bucket algorithm per shard.
//
// Backend limits per shard:
// - 1000 records per second
// - 1 MB per second
//
// With RateLimit=150 (default):
// - 1500 records per second per shard (1000 × 1.5)
// - 1.5 MB per second per shard (1 MB × 1.5)
//
// The default of 150% allows a single producer to completely saturate a shard.
// Set to 100 to match exact shard limits, or lower to reduce throttling risk.
// Minimum value: 1, Maximum value: 9223372036854775807
//
// Default: 150 (matches Java KPL)
// Note: ShardMap and RateLimit are always enabled (matching Java KPL behavior)
RateLimit int
// ShardMap is an optional pre-configured ShardMap for testing purposes.
// If provided, the producer will use this instead of creating a new one.
// Leave nil for normal operation where ShardMap will be automatically initialized.
ShardMap *ShardMap
// ShardMapRefreshTimeout is the maximum time to wait for a ShardMap refresh operation.
// This timeout is used when calling DescribeStream API.
// Default: 30 seconds
ShardMapRefreshTimeout time.Duration
}
Config is the Producer configuration.
type FailureRecord ¶
type FailureRecord struct {
Data []byte
PartitionKey string
// contains filtered or unexported fields
}
Failure record type
type Limiter ¶
type Limiter struct {
// contains filtered or unexported fields
}
Limiter implements per-shard rate limiting using token bucket algorithm. Each shard has independent rate limits for records per second and bytes per second.
func NewLimiter ¶
func NewLimiter(config *LimiterConfig) *Limiter
NewLimiter creates a new Limiter with the given configuration.
func (*Limiter) Allow ¶
Allow checks if the operation is allowed under rate limits without blocking. Returns true if allowed, false if rate limited.
func (*Limiter) GetShardCount ¶
GetShardCount returns the number of shards being tracked.
type LimiterConfig ¶
type LimiterConfig struct {
// RateLimit is the maximum allowed put rate as a percentage of backend limits.
// Backend limits per shard: 1000 records/sec, 1 MB/sec
// Default: 150 (150% = 1500 records/sec, 1.5 MB/sec)
RateLimit int
}
LimiterConfig holds rate limiting configuration.
type LogValue ¶
type LogValue struct {
Name string
Value interface{}
}
LogValue represents a key:value pair used by the Logger interface
type Logger ¶
type Logger interface {
Info(msg string, values ...LogValue)
Error(msg string, err error, values ...LogValue)
}
Logger represents a simple interface used by kinesis-producer to handle logging
type Producer ¶
Producer batches records.
func New ¶
New creates new producer with the given config. Returns an error if the configuration is invalid.
func (*Producer) NotifyFailures ¶
func (p *Producer) NotifyFailures() <-chan *FailureRecord
NotifyFailures registers and return listener to handle undeliverable messages. The incoming struct has a copy of the Data and the PartitionKey along with some error information about why the publishing failed.
func (*Producer) Put ¶
Put `data` using `partitionKey` asynchronously. This method is thread-safe.
Under the covers, the Producer will automatically re-attempt puts in case of transient errors. When unrecoverable error has detected(e.g: trying to put to in a stream that doesn't exist), the message will returned by the Producer. Add a listener with `Producer.NotifyFailures` to handle undeliverable messages.
type Putter ¶
type Putter interface {
PutRecords(context.Context, *k.PutRecordsInput, ...func(*k.Options)) (*k.PutRecordsOutput, error)
}
Putter is the interface that wraps the KinesisAPI.PutRecords method.
type Record ¶
type Record struct {
PartitionKeyIndex *uint64 `protobuf:"varint,1,req,name=partition_key_index,json=partitionKeyIndex" json:"partition_key_index,omitempty"`
ExplicitHashKeyIndex *uint64 `protobuf:"varint,2,opt,name=explicit_hash_key_index,json=explicitHashKeyIndex" json:"explicit_hash_key_index,omitempty"`
Data []byte `protobuf:"bytes,3,req,name=data" json:"data,omitempty"`
Tags []*Tag `protobuf:"bytes,4,rep,name=tags" json:"tags,omitempty"`
// contains filtered or unexported fields
}
func (*Record) Descriptor
deprecated
func (*Record) GetExplicitHashKeyIndex ¶
func (*Record) GetPartitionKeyIndex ¶
func (*Record) ProtoMessage ¶
func (*Record) ProtoMessage()
func (*Record) ProtoReflect ¶
func (x *Record) ProtoReflect() protoreflect.Message
type ShardAggregator ¶
type ShardAggregator struct {
// contains filtered or unexported fields
}
ShardAggregator aggregates records for a single shard. All records in this aggregator will be sent to the same shard.
func NewShardAggregator ¶
func NewShardAggregator(shardID string, config *AggregatorConfig) *ShardAggregator
NewShardAggregator creates a new shard-specific aggregator.
func (*ShardAggregator) Count ¶
func (sa *ShardAggregator) Count() int
Count returns the number of records.
func (*ShardAggregator) Drain ¶
func (sa *ShardAggregator) Drain() (*ktypes.PutRecordsRequestEntry, error)
Drain creates an aggregated PutRecordsRequestEntry and resets the aggregator. Returns nil if there are no records to drain.
func (*ShardAggregator) IsEmpty ¶
func (sa *ShardAggregator) IsEmpty() bool
IsEmpty returns true if the aggregator has no records.
func (*ShardAggregator) Put ¶
func (sa *ShardAggregator) Put(data []byte, partitionKey string) bool
Put adds a record to the aggregator with partition key deduplication. Returns true if the aggregator needs to be flushed after this put.
func (*ShardAggregator) ShardID ¶
func (sa *ShardAggregator) ShardID() string
ShardID returns the shard ID this aggregator is for.
func (*ShardAggregator) ShouldFlush ¶
func (sa *ShardAggregator) ShouldFlush() bool
ShouldFlush checks if the aggregator should be flushed based on size or count limits.
func (*ShardAggregator) Size ¶
func (sa *ShardAggregator) Size() int
Size returns the current size in bytes.
type ShardLimiter ¶
type ShardLimiter struct {
// contains filtered or unexported fields
}
ShardLimiter implements token bucket rate limiting for a single shard.
type ShardMap ¶
type ShardMap struct {
// contains filtered or unexported fields
}
ShardMap maintains a mapping from partition key hashes to shard IDs. It uses an event-driven approach to handle shard splits and merges.
Event-Driven Updates (matching Java KPL behavior): The shard map is invalidated when a mismatch is detected between the predicted shard and the actual shard where a record was sent. This triggers an immediate refresh with exponential backoff retry logic (1s -> 30s).
Unlike the periodic refresh approach, this matches Java KPL's reactive strategy where updates only occur when needed.
func NewShardMap ¶
func NewShardMap(streamName string, client *k.Client, logger Logger, verbose bool, refreshTimeout time.Duration) (*ShardMap, error)
NewShardMap creates a new ShardMap with event-driven updates. The shard map uses an event-driven approach (matching Java KPL) where updates only occur when a mismatch is detected. refreshTimeout specifies the maximum time to wait for DescribeStream API calls.
func (*ShardMap) GetShardCount ¶
GetShardCount returns the current number of open shards.
func (*ShardMap) GetShardID ¶
GetShardID returns the shard ID for a given partition key. It computes the MD5 hash of the partition key and performs a binary search through the sorted shard ranges.
Thread-safety: Uses RLock to allow concurrent reads. The Copy-on-Write pattern in refresh() ensures readers never see partial updates and are only briefly blocked during the atomic pointer swap.
func (*ShardMap) GetShardIDByHash ¶
GetShardIDByHash returns the shard ID for a given hash key (decimal string). This is used to compute the predicted shard ID from ExplicitHashKey.
func (*ShardMap) Invalidate ¶
Invalidate marks the shard map as invalid and triggers an immediate refresh. This is called when a mismatch is detected between the predicted shard and the actual shard where a record was sent.
seenAt: The time when the mismatch was detected predictedShardID: The shard ID that was predicted (optional)
type ShardMapState ¶
type ShardMapState int
ShardMapState represents the current state of the shard map.
const ( ShardMapStateReady ShardMapState = iota // Shard map is ready for use ShardMapStateInvalid // Shard map is invalid and needs refresh ShardMapStateUpdating // Shard map is currently being updated )
type ShardRange ¶
type ShardRange struct {
ShardID string
StartingHashKey *big.Int // 128-bit hash key
EndingHashKey *big.Int // 128-bit hash key
}
ShardRange represents a shard's hash key range.
type StdLogger ¶
StdLogger implements the Logger interface using standard library loggers
type Tag ¶
type Tag struct {
Key *string `protobuf:"bytes,1,req,name=key" json:"key,omitempty"`
Value *string `protobuf:"bytes,2,opt,name=value" json:"value,omitempty"`
// contains filtered or unexported fields
}
func (*Tag) Descriptor
deprecated
func (*Tag) ProtoMessage ¶
func (*Tag) ProtoMessage()
func (*Tag) ProtoReflect ¶
func (x *Tag) ProtoReflect() protoreflect.Message