kcl

README

Amazon Kinesis Client Library for Go

This package provides an interface to the Amazon Kinesis Client Library (KCL) MultiLangDaemon for Golang.

Developers can use the KCL to build distributed applications that process streaming data reliably at scale. The KCL takes care of many of the complex tasks associated with distributed computing, such as load-balancing across multiple instances, responding to instance failures, checkpointing processed records and reacting to changes in stream volume.

This package wraps and manages the interaction with the MultiLangDaemon, which is provided as part of the Amazon KCL for Java so that developers can focus on implementing their record processing logic.

A record processor in Go must implement the RecordProcessor interface and call the function Run.

Documentation

Index

• func Run(processor RecordProcessor)
• type Checkpointer
  • func (cp *Checkpointer) Checkpoint(sequenceNumber *string) error
• type Record
• type RecordProcessor
• type ShutdownType

Functions

func Run

func Run(processor RecordProcessor)

Run ...

Types

type Checkpointer

type Checkpointer struct {
}

Checkpointer ...

func (*Checkpointer) Checkpoint

func (cp *Checkpointer) Checkpoint(sequenceNumber *string) error

Checkpoint ...

type Record

type Record struct {
	Data           string `json:"data"`
	PartitionKey   string `json:"partitionKey"`
	SequenceNumber string `json:"sequenceNumber"`
}

Record ...

type RecordProcessor

type RecordProcessor interface {
	// Initialize is called once by the KCL before any calls to processRecords.
	// Any initialization logic for record processing can go here.
	Initialize(string) error

	// ProcessRecords is called by KCL with a list of records to be processed
	// and checkpointed. The checkpointer can optionally be used to checkpoint
	// a particular sequence number (from a record).
	ProcessRecords([]*Record, *Checkpointer) error

	// ShutdownRequested is called by KCL to indicate that this record processor
	// should shut down. After the shutdown requested operation is complete,
	// there will not be any more calls to any other functions of this record
	// processor. Clients should not attempt to checkpoint because the lease has
	// been lost by this Worker.
	ShutdownRequested(*Checkpointer) error

	// LeaseLost is called by the KCL to indicate that this record processor should shut down.
	// After the lease lost operation is complete, there will not be any more calls to
	// any other functions of this record processor. Clients should not attempt to
	// checkpoint because the lease has been lost by this Worker.
	LeaseLost() error

	// ShardEnded is called by the KCL to indicate that this record processor should shutdown.
	// After the shard ended operation is complete, there will not be any more calls to
	// any other functions of this record processor. Clients are required to checkpoint
	// at this time. This indicates that the current record processor has finished
	// processing and new record processors for the children will be created.
	ShardEnded(*Checkpointer) error
}

RecordProcessor ...

type ShutdownType

type ShutdownType int

ShutdownType ...

const (

	// GracefulShutdown ...
	GracefulShutdown ShutdownType

	// ZombieShutdown ...
	ZombieShutdown
)