Documentation
¶
Overview ¶
Package kafka provides a thin, testable wrapper around IBM Sarama for producers and consumer groups.
Design constraints (for use in application frameworks):
- No viper, no global singletons, no log.Fatal inside the library.
- Constructors return interfaces (Producer, GroupConsumer) for mocking.
- ConfigDefault() for connection Config; ProducerConfigDefault() for producer tuning.
- Consumers use NewConsumer + Start(ctx) + Close(); start them from main or a dedicated cmd process.
Producers:
- NewSyncProducer — blocking send per call.
- NewAsyncProducer — pipelined send with SendMessage still waiting for broker ack (goroutine-safe).
Consumers:
- NewConsumer / NewConsumerWithDLQ — worker pool, retries, optional dead-letter topic.
Index ¶
- Constants
- Variables
- type AsyncProducer
- type Config
- type Consumer
- func (c *Consumer) Cleanup(_ sarama.ConsumerGroupSession) error
- func (c *Consumer) Close() error
- func (c *Consumer) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error
- func (c *Consumer) Setup(_ sarama.ConsumerGroupSession) error
- func (c *Consumer) Start(ctx context.Context) error
- type ConsumerConfig
- type ConsumerHandler
- type ConsumerHooks
- type DLQEnvelope
- type GroupConsumer
- type Producer
- type ProducerConfig
- type SyncProducer
Examples ¶
Constants ¶
const ( InitialOffsetNewest = "newest" InitialOffsetOldest = "oldest" )
Initial offset policy names (new consumer group or partition without committed offset).
const ( RebalanceRange = "range" RebalanceSticky = "sticky" RebalanceRoundRobin = "roundrobin" )
Consumer group rebalance strategy names.
const ( CompressionNone = "none" CompressionGzip = "gzip" CompressionSnappy = "snappy" CompressionLZ4 = "lz4" CompressionZstd = "zstd" )
Compression codec names for ProducerConfig.Compression.
Variables ¶
var ( // ErrProducerClosed is returned when SendMessage is called after Close or during shutdown. ErrProducerClosed = errors.New("kafka: producer closed") // ErrInvalidConfig indicates validation failed on Config, ProducerConfig, or ConsumerConfig. ErrInvalidConfig = errors.New("kafka: invalid config") // ErrHandlerNil is returned when NewConsumer is called with a nil handler. ErrHandlerNil = errors.New("kafka: handler is nil") // ErrMetadataSet is returned when AsyncProducer.SendMessage is called with non-nil ProducerMessage.Metadata. ErrMetadataSet = errors.New("kafka: ProducerMessage.Metadata is reserved by AsyncProducer") )
Functions ¶
This section is empty.
Types ¶
type AsyncProducer ¶
type AsyncProducer struct {
// contains filtered or unexported fields
}
AsyncProducer wraps sarama.AsyncProducer with a synchronous SendMessage API.
SendMessage is goroutine-safe, blocks until the broker acks (or returns an error), and is not fire-and-forget. Use a shared instance from many HTTP handlers for throughput.
ProducerMessage.Metadata is reserved; do not set it on messages passed to SendMessage.
Always call Close on shutdown to flush in-flight messages.
func (*AsyncProducer) Close ¶
func (ap *AsyncProducer) Close() error
Close shuts down the producer and waits for background goroutines.
func (*AsyncProducer) SendMessage ¶
func (ap *AsyncProducer) SendMessage(ctx context.Context, msg *sarama.ProducerMessage) (int32, int64, error)
SendMessage blocks until ack, error, ctx cancellation, or producer closed.
type Config ¶
type Config struct {
// Brokers is the list of broker addresses (required), e.g. []string{"host1:9092"}.
Brokers []string
// KafkaVersion is the broker protocol version string (e.g. "3.6.0"). Empty defaults to V4_1_0_0.
KafkaVersion string
// SASLEnable enables SASL authentication (plaintext mechanism only).
SASLEnable bool
// SASLUser is the SASL username when SASLEnable is true.
SASLUser string
// SASLPassword is the SASL password when SASLEnable is true.
SASLPassword string
// TLSEnable enables TLS. TLSCA alone (server verification) is sufficient;
// TLSCert and TLSKey add mutual TLS and must be set together.
TLSEnable bool
// TLSInsecureSkipVerify skips server certificate verification (dev/debug only).
TLSInsecureSkipVerify bool
// TLSCert is the client certificate PEM for mutual TLS.
TLSCert string
// TLSKey is the client private key PEM.
TLSKey string
// TLSCA is the root CA PEM used to verify broker certificates.
TLSCA string
}
Config holds connection settings (brokers, SASL, TLS) shared by producers and consumers.
func ConfigDefault ¶
func ConfigDefault() Config
ConfigDefault returns connection defaults; Brokers must still be set before use.
func (Config) BuildTLS ¶
BuildTLS returns *tls.Config when TLSEnable is true, otherwise nil.
TLSCA, when set, becomes the root pool (CA-only is valid for server-auth TLS). TLSCert and TLSKey must be set together for mutual TLS. With none of them set the system root pool is used.
func (Config) SaramaVersion ¶
func (c Config) SaramaVersion() sarama.KafkaVersion
SaramaVersion returns the parsed Kafka protocol version for Sarama.
func (Config) ValidateBrokers ¶
ValidateBrokers ensures brokers is non-empty.
type Consumer ¶
type Consumer struct {
// contains filtered or unexported fields
}
Consumer wraps a Sarama consumer group with worker-pool concurrency and retries.
func (*Consumer) Cleanup ¶
func (c *Consumer) Cleanup(_ sarama.ConsumerGroupSession) error
Cleanup implements sarama.ConsumerGroupHandler.
func (*Consumer) ConsumeClaim ¶
func (c *Consumer) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error
ConsumeClaim implements sarama.ConsumerGroupHandler.
type ConsumerConfig ¶
type ConsumerConfig struct {
// GroupID is the consumer group id (required).
GroupID string
// Topics lists topics to subscribe to (required).
Topics []string
// Concurrency is the number of workers per partition claim (default 1).
Concurrency int
// MaxRetries is how many times to retry the handler after the first failure (default allows MaxRetries+1 total attempts).
MaxRetries int
// RetryDelay is the pause between handler retries on the same message.
RetryDelay time.Duration
// DLQTopic is the dead-letter topic; requires a non-nil dlq producer in NewConsumerWithDLQ.
DLQTopic string
// BootstrapMaxRetries is how many extra attempts to create the consumer group (0 = one try).
BootstrapMaxRetries int
// BootstrapRetryDelay is the pause between bootstrap retries.
BootstrapRetryDelay time.Duration
// ConsumeRetryDelay is the pause after Consume returns an error before retrying Start loop.
ConsumeRetryDelay time.Duration
// WorkerShutdownTimeout is how long to wait for workers after a claim ends (default 10s).
WorkerShutdownTimeout time.Duration
// InitialOffset is newest or oldest when no committed offset exists.
InitialOffset string
// RebalanceStrategy is range, sticky, or roundrobin.
RebalanceStrategy string
// Hooks optional observability callbacks.
Hooks ConsumerHooks
}
ConsumerConfig tunes consumer group behavior.
func (*ConsumerConfig) Validate ¶
func (c *ConsumerConfig) Validate() error
Validate fills defaults and validates consumer config.
type ConsumerHandler ¶
type ConsumerHandler interface {
HandleMessage(ctx context.Context, msg *sarama.ConsumerMessage) error
}
ConsumerHandler handles a single Kafka message.
type ConsumerHooks ¶
type ConsumerHooks struct {
// OnHandlerError is called when the handler returns an error; attempt starts at 1.
OnHandlerError func(ctx context.Context, msg *sarama.ConsumerMessage, err error, attempt int)
// OnPoison is called when MaxRetries is exceeded, before DLQ or mark.
OnPoison func(ctx context.Context, msg *sarama.ConsumerMessage, err error)
// OnDLQFailed is called when publishing to the DLQ topic fails; the offset is not committed and the claim ends so the message is redelivered.
OnDLQFailed func(ctx context.Context, msg *sarama.ConsumerMessage, dlqErr error)
// OnGroupError is called for errors from ConsumerGroup.Errors() (rebalance, network, etc.).
OnGroupError func(ctx context.Context, err error)
// OnWorkerShutdownTimeout is called when workers do not finish within WorkerShutdownTimeout after claim ends.
OnWorkerShutdownTimeout func(ctx context.Context)
}
ConsumerHooks optional callbacks for observability (logging, metrics). Nil funcs are skipped.
type DLQEnvelope ¶
type DLQEnvelope struct {
// OriginalTopic is the source topic of the failed message.
OriginalTopic string `json:"original_topic"`
// OriginalPartition is the source partition.
OriginalPartition int32 `json:"original_partition"`
// OriginalOffset is the source offset.
OriginalOffset int64 `json:"original_offset"`
// OriginalKey is the source message key.
OriginalKey []byte `json:"original_key,omitempty"`
// OriginalValue is the source message value.
OriginalValue []byte `json:"original_value,omitempty"`
// Error is the handler error string.
Error string `json:"error"`
// FailedAt is the Unix timestamp when the handler failed.
FailedAt int64 `json:"failed_at_unix"`
}
DLQEnvelope is the JSON payload written to a dead-letter topic.
type GroupConsumer ¶
GroupConsumer is the runtime abstraction for a Kafka consumer group (mock-friendly).
func NewConsumer ¶
func NewConsumer(ctx context.Context, cfg Config, consumerConfig ConsumerConfig, handler ConsumerHandler) (GroupConsumer, error)
NewConsumer creates a consumer group without DLQ producer.
func NewConsumerWithDLQ ¶
func NewConsumerWithDLQ(ctx context.Context, cfg Config, consumerConfig ConsumerConfig, handler ConsumerHandler, dlqProducer Producer) (GroupConsumer, error)
NewConsumerWithDLQ creates a consumer group; dlqProducer is required when DLQTopic is set. ctx bounds the connect and the bootstrap retries (including their backoff).
type Producer ¶
type Producer interface {
SendMessage(ctx context.Context, msg *sarama.ProducerMessage) (partition int32, offset int64, err error)
Close() error
}
Producer is the recommended producer abstraction (sync or async wrapper).
SendMessage honours ctx before the message is handed to Sarama; the async implementation also aborts while waiting for the broker ack.
func NewAsyncProducer ¶
NewAsyncProducer creates an async producer with the given connection and producer config. ctx bounds the initial broker connection.
Example ¶
package main
import (
"context"
"github.com/IBM/sarama"
"github.com/andywang2011/go-kafka"
)
func main() {
cfg := kafka.Config{Brokers: []string{"127.0.0.1:9092"}}
pc := kafka.ProducerConfigDefault()
p, err := kafka.NewAsyncProducer(context.Background(), cfg, pc)
if err != nil {
return
}
defer p.Close()
_, _, _ = p.SendMessage(context.Background(), &sarama.ProducerMessage{
Topic: "orders",
Key: sarama.StringEncoder("user-1"),
Value: sarama.ByteEncoder([]byte(`{"id":1}`)),
})
}
Output:
func NewProducer ¶
NewProducer is deprecated; use NewSyncProducer with ProducerConfigDefault.
func NewSyncProducer ¶
NewSyncProducer creates a synchronous Kafka producer. ctx bounds the initial broker connection.
type ProducerConfig ¶
type ProducerConfig struct {
// RequiredAcks is the broker ack level (default WaitForAll).
RequiredAcks sarama.RequiredAcks
// Compression is none, gzip, snappy, lz4, or zstd.
Compression string
// ChannelBufferSize is the async producer input buffer size (default 1024).
ChannelBufferSize int
// FlushFrequency enables time-based batch flush when > 0.
FlushFrequency time.Duration
// FlushMessages enables count-based batch flush when > 0.
FlushMessages int
// Idempotent enables the idempotent producer (requires Kafka 0.11+).
Idempotent bool
}
ProducerConfig tunes producer behavior (acks, compression, buffering).
func ProducerConfigDefault ¶
func ProducerConfigDefault() ProducerConfig
ProducerConfigDefault returns recommended defaults for production use.
func (*ProducerConfig) Validate ¶
func (pc *ProducerConfig) Validate() error
Validate fills defaults and checks producer config.
type SyncProducer ¶
type SyncProducer = sarama.SyncProducer
SyncProducer is the raw Sarama sync producer type alias for advanced use and mocks.