Documentation
¶
Index ¶
- Variables
- type AllGroup
- type CloseableGroup
- type Consumer
- type Counter
- type Group
- type Logger
- type Option
- func WithAggregation(a bool) Option
- func WithClient(client kinesisClient) Option
- func WithCounter(counter Counter) Option
- func WithGetRecordsOptions(opts ...func(*kinesis.Options)) Option
- func WithGroup(group Group) Option
- func WithLogger(logger Logger) Option
- func WithMaxRecords(n int64) Option
- func WithScanInterval(d time.Duration) Option
- func WithShardClosedHandler(h ShardClosedHandler) Option
- func WithShardIteratorType(t string) Option
- func WithStore(store Store) Option
- func WithTimestamp(t time.Time) Option
- type Record
- type ScanBatchFunc
- type ScanBatchOption
- type ScanFunc
- type ShardClosedHandler
- type Store
Constants ¶
This section is empty.
Variables ¶
var ErrSkipCheckpoint = errors.New("skip checkpoint")
ErrSkipCheckpoint is used as a return value from ScanFunc to indicate that the current checkpoint should be skipped. It is not returned as an error by any function.
Functions ¶
This section is empty.
Types ¶
type AllGroup ¶ added in v0.3.3
type AllGroup struct {
Store
// contains filtered or unexported fields
}
AllGroup is used to consume all shards from a single consumer. It caches a local list of the shards we are already processing and routinely polls the stream looking for new shards to process.
func NewAllGroup ¶ added in v0.3.3
NewAllGroup returns an initialized AllGroup for consuming all shards on a stream
func (*AllGroup) CloseShard ¶ added in v0.3.6
type CloseableGroup ¶ added in v0.3.6
type Consumer ¶
type Consumer struct {
// contains filtered or unexported fields
}
Consumer wraps the interaction with the Kinesis stream
func New ¶
New creates a kinesis consumer with default settings. Use Option to override any of the optional attributes.
func (*Consumer) Scan ¶
Scan launches a goroutine to process each of the shards in the stream. The ScanFunc is passed through to each of the goroutines and called with each message pulled from the stream.
func (*Consumer) ScanBatch ¶ added in v0.4.1
func (c *Consumer) ScanBatch(ctx context.Context, fn ScanBatchFunc, opts ...ScanBatchOption) error
ScanBatch scans all shards and delivers buffered records to a batch callback. Existing Scan behavior remains unchanged and this method is opt-in.
Checkpoint semantics: - Each shard is checkpointed only after its batch callback succeeds. - On callback error, scanning stops and that batch is not checkpointed.
type Group ¶ added in v0.3.3
type Group interface {
Start(ctx context.Context, shardc chan types.Shard) error
GetCheckpoint(streamName, shardID string) (string, error)
SetCheckpoint(streamName, shardID, sequenceNumber string) error
}
Group interface used to manage which shard to process
type Logger ¶
type Logger interface {
Log(...interface{})
}
A Logger is a minimal interface to as a adaptor for external logging library to consumer
type Option ¶
type Option func(*Consumer)
Option is used to override defaults when creating a new Consumer
func WithAggregation ¶ added in v0.3.5
func WithClient ¶
func WithClient(client kinesisClient) Option
WithClient overrides the default client
func WithCounter ¶
WithCounter overrides the default counter
func WithGetRecordsOptions ¶ added in v0.4.0
WithGetRecordsOptions passes the given option functions to the kinesis client's GetRecords call
func WithMaxRecords ¶ added in v0.3.4
WithMaxRecords overrides the maximum number of records to be returned in a single GetRecords call for the consumer (specify a value of up to 10,000)
func WithScanInterval ¶ added in v0.3.4
WithScanInterval overrides the scan interval for the consumer
func WithShardClosedHandler ¶ added in v0.3.5
func WithShardClosedHandler(h ShardClosedHandler) Option
func WithShardIteratorType ¶ added in v0.3.1
WithShardIteratorType overrides the starting point for the consumer
func WithTimestamp ¶ added in v0.3.3
WithTimestamp overrides the starting point for the consumer
type Record ¶
Record wraps the record returned from the Kinesis library and extends to include the shard id.
type ScanBatchFunc ¶ added in v0.4.1
ScanBatchFunc is called with buffered records from a shard. Checkpoint advances only after this callback returns nil.
type ScanBatchOption ¶ added in v0.4.1
type ScanBatchOption func(*scanBatchConfig)
ScanBatchOption customizes batch behavior for ScanBatch.
func WithBatchFlushInterval ¶ added in v0.4.1
func WithBatchFlushInterval(d time.Duration) ScanBatchOption
WithBatchFlushInterval sets how often pending batches are flushed. A non-positive duration disables periodic flushing.
func WithBatchMaxSize ¶ added in v0.4.1
func WithBatchMaxSize(n int) ScanBatchOption
WithBatchMaxSize sets the per-shard max buffered record count before flush.
type ScanFunc ¶ added in v0.3.2
ScanFunc is the type of the function called for each message read from the stream. The record argument contains the original record returned from the AWS Kinesis library. If an error is returned, scanning stops. The sole exception is when the function returns the special value ErrSkipCheckpoint.
type ShardClosedHandler ¶ added in v0.3.5
ShardClosedHandler is a handler that will be called when the consumer has reached the end of a closed shard. No more records for that shard will be provided by the consumer. An error can be returned to stop the consumer.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
examples
|
|
|
consumer
command
|
|
|
consumer-dynamodb
command
|
|
|
consumer-group-ddb
command
|
|
|
consumer-mysql
command
|
|
|
consumer-postgres
command
|
|
|
consumer-redis
command
|
|
|
producer
command
|
|
|
group
|
|
|
store
|
|
|
memory
The memory store provides a store that can be used for testing and single-threaded applications.
|
The memory store provides a store that can be used for testing and single-threaded applications. |