kafka

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 7, 2026 License: MIT Imports: 11 Imported by: 0

README

go-kafka

Thin, testable Kafka client for Go services, built on IBM Sarama.

Install

go get github.com/andywang2011/go-kafka@v0.1.0

Quick start

cfg := kafka.Config{Brokers: []string{"127.0.0.1:9092"}}
pc := kafka.ProducerConfigDefault()

p, err := kafka.NewAsyncProducer(ctx, cfg, pc)
if err != nil {
	// handle
}
defer p.Close()

_, _, err = p.SendMessage(ctx, &sarama.ProducerMessage{
	Topic: "orders",
	Key:   sarama.StringEncoder("user-1"),
	Value: sarama.ByteEncoder([]byte(`{"id":1}`)),
})

Examples

export KAFKA_BROKERS=127.0.0.1:9092
cd go-kafka/examples/producer && go run .
cd go-kafka/examples/consumer && go run .

License

MIT — see LICENSE.

Documentation

Overview

Package kafka provides a thin, testable wrapper around IBM Sarama for producers and consumer groups.

Design constraints (for use in application frameworks):

  • No viper, no global singletons, no log.Fatal inside the library.
  • Constructors return interfaces (Producer, GroupConsumer) for mocking.
  • ConfigDefault() for connection Config; ProducerConfigDefault() for producer tuning.
  • Consumers use NewConsumer + Start(ctx) + Close(); start them from main or a dedicated cmd process.

Producers:

  • NewSyncProducer — blocking send per call.
  • NewAsyncProducer — pipelined send with SendMessage still waiting for broker ack (goroutine-safe).

Consumers:

  • NewConsumer / NewConsumerWithDLQ — worker pool, retries, optional dead-letter topic.

Index

Examples

Constants

View Source
const (
	InitialOffsetNewest = "newest"
	InitialOffsetOldest = "oldest"
)

Initial offset policy names (new consumer group or partition without committed offset).

View Source
const (
	RebalanceRange      = "range"
	RebalanceSticky     = "sticky"
	RebalanceRoundRobin = "roundrobin"
)

Consumer group rebalance strategy names.

View Source
const (
	CompressionNone   = "none"
	CompressionGzip   = "gzip"
	CompressionSnappy = "snappy"
	CompressionLZ4    = "lz4"
	CompressionZstd   = "zstd"
)

Compression codec names for ProducerConfig.Compression.

Variables

View Source
var (
	// ErrProducerClosed is returned when SendMessage is called after Close or during shutdown.
	ErrProducerClosed = errors.New("kafka: producer closed")

	// ErrInvalidConfig indicates validation failed on Config, ProducerConfig, or ConsumerConfig.
	ErrInvalidConfig = errors.New("kafka: invalid config")

	// ErrHandlerNil is returned when NewConsumer is called with a nil handler.
	ErrHandlerNil = errors.New("kafka: handler is nil")

	// ErrMetadataSet is returned when AsyncProducer.SendMessage is called with non-nil ProducerMessage.Metadata.
	ErrMetadataSet = errors.New("kafka: ProducerMessage.Metadata is reserved by AsyncProducer")
)

Functions

This section is empty.

Types

type AsyncProducer

type AsyncProducer struct {
	// contains filtered or unexported fields
}

AsyncProducer wraps sarama.AsyncProducer with a synchronous SendMessage API.

SendMessage is goroutine-safe, blocks until the broker acks (or returns an error), and is not fire-and-forget. Use a shared instance from many HTTP handlers for throughput.

ProducerMessage.Metadata is reserved; do not set it on messages passed to SendMessage.

Always call Close on shutdown to flush in-flight messages.

func (*AsyncProducer) Close

func (ap *AsyncProducer) Close() error

Close shuts down the producer and waits for background goroutines.

func (*AsyncProducer) SendMessage

func (ap *AsyncProducer) SendMessage(ctx context.Context, msg *sarama.ProducerMessage) (int32, int64, error)

SendMessage blocks until ack, error, ctx cancellation, or producer closed.

type Config

type Config struct {
	// Brokers is the list of broker addresses (required), e.g. []string{"host1:9092"}.
	Brokers []string

	// KafkaVersion is the broker protocol version string (e.g. "3.6.0"). Empty defaults to V4_1_0_0.
	KafkaVersion string

	// SASLEnable enables SASL authentication (plaintext mechanism only).
	SASLEnable bool
	// SASLUser is the SASL username when SASLEnable is true.
	SASLUser string
	// SASLPassword is the SASL password when SASLEnable is true.
	SASLPassword string

	// TLSEnable enables TLS. TLSCA alone (server verification) is sufficient;
	// TLSCert and TLSKey add mutual TLS and must be set together.
	TLSEnable bool
	// TLSInsecureSkipVerify skips server certificate verification (dev/debug only).
	TLSInsecureSkipVerify bool
	// TLSCert is the client certificate PEM for mutual TLS.
	TLSCert string
	// TLSKey is the client private key PEM.
	TLSKey string
	// TLSCA is the root CA PEM used to verify broker certificates.
	TLSCA string
}

Config holds connection settings (brokers, SASL, TLS) shared by producers and consumers.

func ConfigDefault

func ConfigDefault() Config

ConfigDefault returns connection defaults; Brokers must still be set before use.

func (Config) BuildTLS

func (c Config) BuildTLS() (*tls.Config, error)

BuildTLS returns *tls.Config when TLSEnable is true, otherwise nil.

TLSCA, when set, becomes the root pool (CA-only is valid for server-auth TLS). TLSCert and TLSKey must be set together for mutual TLS. With none of them set the system root pool is used.

func (Config) SaramaVersion

func (c Config) SaramaVersion() sarama.KafkaVersion

SaramaVersion returns the parsed Kafka protocol version for Sarama.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks connection config.

func (Config) ValidateBrokers

func (c Config) ValidateBrokers() error

ValidateBrokers ensures brokers is non-empty.

type Consumer

type Consumer struct {
	// contains filtered or unexported fields
}

Consumer wraps a Sarama consumer group with worker-pool concurrency and retries.

func (*Consumer) Cleanup

func (c *Consumer) Cleanup(_ sarama.ConsumerGroupSession) error

Cleanup implements sarama.ConsumerGroupHandler.

func (*Consumer) Close

func (c *Consumer) Close() error

Close closes the consumer group; safe to call multiple times.

func (*Consumer) ConsumeClaim

func (c *Consumer) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error

ConsumeClaim implements sarama.ConsumerGroupHandler.

func (*Consumer) Setup

Setup implements sarama.ConsumerGroupHandler.

func (*Consumer) Start

func (c *Consumer) Start(ctx context.Context) error

Start blocks until ctx is cancelled or the consumer group is closed.

type ConsumerConfig

type ConsumerConfig struct {
	// GroupID is the consumer group id (required).
	GroupID string
	// Topics lists topics to subscribe to (required).
	Topics []string
	// Concurrency is the number of workers per partition claim (default 1).
	Concurrency int
	// MaxRetries is how many times to retry the handler after the first failure (default allows MaxRetries+1 total attempts).
	MaxRetries int
	// RetryDelay is the pause between handler retries on the same message.
	RetryDelay time.Duration

	// DLQTopic is the dead-letter topic; requires a non-nil dlq producer in NewConsumerWithDLQ.
	DLQTopic string

	// BootstrapMaxRetries is how many extra attempts to create the consumer group (0 = one try).
	BootstrapMaxRetries int
	// BootstrapRetryDelay is the pause between bootstrap retries.
	BootstrapRetryDelay time.Duration

	// ConsumeRetryDelay is the pause after Consume returns an error before retrying Start loop.
	ConsumeRetryDelay time.Duration

	// WorkerShutdownTimeout is how long to wait for workers after a claim ends (default 10s).
	WorkerShutdownTimeout time.Duration

	// InitialOffset is newest or oldest when no committed offset exists.
	InitialOffset string
	// RebalanceStrategy is range, sticky, or roundrobin.
	RebalanceStrategy string

	// Hooks optional observability callbacks.
	Hooks ConsumerHooks
}

ConsumerConfig tunes consumer group behavior.

func (*ConsumerConfig) Validate

func (c *ConsumerConfig) Validate() error

Validate fills defaults and validates consumer config.

type ConsumerHandler

type ConsumerHandler interface {
	HandleMessage(ctx context.Context, msg *sarama.ConsumerMessage) error
}

ConsumerHandler handles a single Kafka message.

type ConsumerHooks

type ConsumerHooks struct {
	// OnHandlerError is called when the handler returns an error; attempt starts at 1.
	OnHandlerError func(ctx context.Context, msg *sarama.ConsumerMessage, err error, attempt int)
	// OnPoison is called when MaxRetries is exceeded, before DLQ or mark.
	OnPoison func(ctx context.Context, msg *sarama.ConsumerMessage, err error)
	// OnDLQFailed is called when publishing to the DLQ topic fails; the offset is not committed and the claim ends so the message is redelivered.
	OnDLQFailed func(ctx context.Context, msg *sarama.ConsumerMessage, dlqErr error)
	// OnGroupError is called for errors from ConsumerGroup.Errors() (rebalance, network, etc.).
	OnGroupError func(ctx context.Context, err error)
	// OnWorkerShutdownTimeout is called when workers do not finish within WorkerShutdownTimeout after claim ends.
	OnWorkerShutdownTimeout func(ctx context.Context)
}

ConsumerHooks optional callbacks for observability (logging, metrics). Nil funcs are skipped.

type DLQEnvelope

type DLQEnvelope struct {
	// OriginalTopic is the source topic of the failed message.
	OriginalTopic string `json:"original_topic"`
	// OriginalPartition is the source partition.
	OriginalPartition int32 `json:"original_partition"`
	// OriginalOffset is the source offset.
	OriginalOffset int64 `json:"original_offset"`
	// OriginalKey is the source message key.
	OriginalKey []byte `json:"original_key,omitempty"`
	// OriginalValue is the source message value.
	OriginalValue []byte `json:"original_value,omitempty"`
	// Error is the handler error string.
	Error string `json:"error"`
	// FailedAt is the Unix timestamp when the handler failed.
	FailedAt int64 `json:"failed_at_unix"`
}

DLQEnvelope is the JSON payload written to a dead-letter topic.

type GroupConsumer

type GroupConsumer interface {
	Start(ctx context.Context) error
	Close() error
}

GroupConsumer is the runtime abstraction for a Kafka consumer group (mock-friendly).

func NewConsumer

func NewConsumer(ctx context.Context, cfg Config, consumerConfig ConsumerConfig, handler ConsumerHandler) (GroupConsumer, error)

NewConsumer creates a consumer group without DLQ producer.

func NewConsumerWithDLQ

func NewConsumerWithDLQ(ctx context.Context, cfg Config, consumerConfig ConsumerConfig, handler ConsumerHandler, dlqProducer Producer) (GroupConsumer, error)

NewConsumerWithDLQ creates a consumer group; dlqProducer is required when DLQTopic is set. ctx bounds the connect and the bootstrap retries (including their backoff).

type Producer

type Producer interface {
	SendMessage(ctx context.Context, msg *sarama.ProducerMessage) (partition int32, offset int64, err error)
	Close() error
}

Producer is the recommended producer abstraction (sync or async wrapper).

SendMessage honours ctx before the message is handed to Sarama; the async implementation also aborts while waiting for the broker ack.

func NewAsyncProducer

func NewAsyncProducer(ctx context.Context, cfg Config, pc ProducerConfig) (Producer, error)

NewAsyncProducer creates an async producer with the given connection and producer config. ctx bounds the initial broker connection.

Example
package main

import (
	"context"
	"github.com/IBM/sarama"

	"github.com/andywang2011/go-kafka"
)

func main() {
	cfg := kafka.Config{Brokers: []string{"127.0.0.1:9092"}}
	pc := kafka.ProducerConfigDefault()

	p, err := kafka.NewAsyncProducer(context.Background(), cfg, pc)
	if err != nil {
		return
	}
	defer p.Close()

	_, _, _ = p.SendMessage(context.Background(), &sarama.ProducerMessage{
		Topic: "orders",
		Key:   sarama.StringEncoder("user-1"),
		Value: sarama.ByteEncoder([]byte(`{"id":1}`)),
	})
}

func NewProducer

func NewProducer(ctx context.Context, cfg Config) (Producer, error)

NewProducer is deprecated; use NewSyncProducer with ProducerConfigDefault.

func NewSyncProducer

func NewSyncProducer(ctx context.Context, cfg Config, pc ProducerConfig) (Producer, error)

NewSyncProducer creates a synchronous Kafka producer. ctx bounds the initial broker connection.

type ProducerConfig

type ProducerConfig struct {
	// RequiredAcks is the broker ack level (default WaitForAll).
	RequiredAcks sarama.RequiredAcks
	// Compression is none, gzip, snappy, lz4, or zstd.
	Compression string
	// ChannelBufferSize is the async producer input buffer size (default 1024).
	ChannelBufferSize int
	// FlushFrequency enables time-based batch flush when > 0.
	FlushFrequency time.Duration
	// FlushMessages enables count-based batch flush when > 0.
	FlushMessages int
	// Idempotent enables the idempotent producer (requires Kafka 0.11+).
	Idempotent bool
}

ProducerConfig tunes producer behavior (acks, compression, buffering).

func ProducerConfigDefault

func ProducerConfigDefault() ProducerConfig

ProducerConfigDefault returns recommended defaults for production use.

func (*ProducerConfig) Validate

func (pc *ProducerConfig) Validate() error

Validate fills defaults and checks producer config.

type SyncProducer

type SyncProducer = sarama.SyncProducer

SyncProducer is the raw Sarama sync producer type alias for advanced use and mocks.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL