Documentation
¶
Overview ¶
Package kafka provides synchronous and asynchronous Kafka publishing and subscription methods.
Messages may be any JSON-serializable value. Byte slices are transported as raw payloads. Structs with exported Headers and Payload fields retain the envelope format: both parts use JSON, and JSON-tagged fields from the Headers struct are transported as Kafka headers. Other message forms are transported without Kafka headers.
The client supports producer-only, consumer-only, and mixed modes. All core components are exposed as interfaces for testing.
Index ¶
- Constants
- Variables
- func BatchErrors(err error) ([]error, bool)
- func FromBrokerMessage(msg any, target any) error
- type AcknowledgeableBrokerMessage
- type BatchPublishResult
- type Broker
- func (b *Broker) Close(ctx context.Context)
- func (b *Broker) Connect(_ context.Context) error
- func (b *Broker) Publish(ctx context.Context, topic string, msg any) error
- func (b *Broker) PublishAsync(ctx context.Context, topic string, msg any) PublishResult
- func (b *Broker) PublishBatch(ctx context.Context, topic string, msgs any) error
- func (b *Broker) PublishBatchAsync(ctx context.Context, topic string, msgs any) BatchPublishResult
- func (b *Broker) Subscribe(ctx context.Context, topic string, ...) (<-chan struct{}, error)
- func (b *Broker) Unsubscribe(ctx context.Context, topic string) error
- type BrokerMessage
- type Config
- type Controller
- type ControllerAdapter
- type ControllerFactory
- type ControllerWriter
- type Mode
- type PublishError
- type PublishResult
- type SaramaFactory
- type Subscription
- type SubscriptionAdapter
Constants ¶
const ( // DefaultMode is the default broker usage mode. DefaultMode Mode = ModeMixed // DefaultMaxBytes is the default maximum Kafka message size. DefaultMaxBytes = 10 * 1024 * 1024 // DefaultTopicCreateAttempts disables automatic topic creation by default. DefaultTopicCreateAttempts = 0 // DefaultTopicCreateBackoff is the default delay between topic creation attempts. DefaultTopicCreateBackoff = 100 * time.Millisecond // DefaultBatchSize is the default number of messages buffered in a producer batch. DefaultBatchSize = 100 // DefaultBatchBytes is the default producer batch size limit in bytes. DefaultBatchBytes int64 = 1024 * 1024 // DefaultBatchTimeout is the default maximum wait for an incomplete producer batch. DefaultBatchTimeout = 10 * time.Millisecond // DefaultNackBackoff is the default delay before retrying a negatively acknowledged message. DefaultNackBackoff = time.Second // DefaultAutoCommit disables consumer offset auto commit by default. DefaultAutoCommit = false // DefaultConnectionTest enables the connection check by default. DefaultConnectionTest = true )
Variables ¶
var ErrWriterClosed = errors.New("kafka writer closed")
ErrWriterClosed is returned by Publish when the producer has been closed.
Functions ¶
func BatchErrors ¶
BatchErrors returns the per-message errors of a batch publish, aligned with the batch that was passed to PublishBatch.
func FromBrokerMessage ¶
FromBrokerMessage deserializes a message received by a Subscribe handler into target. Target must be a non-nil pointer.
JSON is used for arbitrary values. A byte slice receives the raw Kafka payload. An envelope message with exported Headers and Payload fields decodes the payload and Kafka headers separately.
Types ¶
type AcknowledgeableBrokerMessage ¶
type AcknowledgeableBrokerMessage struct {
BrokerMessage
// contains filtered or unexported fields
}
AcknowledgeableBrokerMessage is a transport message with delivery control. It is exposed for controller and subscription test adapters; application handlers receive a *BrokerMessage as an opaque any value.
func NewAcknowledgeableBrokerMessage ¶
func NewAcknowledgeableBrokerMessage( message BrokerMessage, acknowledgment messageAcknowledgment, ) AcknowledgeableBrokerMessage
NewAcknowledgeableBrokerMessage attaches delivery control to a broker message. It is intended for custom controller and subscription adapters.
func (AcknowledgeableBrokerMessage) Ack ¶
func (m AcknowledgeableBrokerMessage) Ack()
Ack acknowledges the delivery when acknowledgment control is present.
func (AcknowledgeableBrokerMessage) Nak ¶
func (m AcknowledgeableBrokerMessage) Nak()
Nak rejects the delivery when acknowledgment control is present.
type BatchPublishResult ¶
type BatchPublishResult interface {
PublishResult
MessageErrors() []error
}
BatchPublishResult reports completion of an asynchronous batch publication. MessageErrors must only be inspected after Done is closed. It returns one error per input message when individual outcomes are available, or nil when the operation failed as a whole.
type Broker ¶
type Broker struct {
// contains filtered or unexported fields
}
Broker is a client for interacting with Kafka.
func (*Broker) Publish ¶
Publish sends the message to the given topic. Broker controller should be connected - returns error otherwise. If the topic does not exist, it is created automatically unless TopicCreateAttempts is 0.
Any JSON-serializable value can be published. Byte slices are sent as raw payloads. A struct with exported Headers and Payload fields uses the envelope format: Payload is JSON encoded as the Kafka value, and JSON-tagged fields from the Headers struct become Kafka headers. Other message forms are published without Kafka headers.
Example envelope message:
type HeadersType struct{
// Some serializable struct
}
type PayloadType struct{
// Some serializable struct
}
type Message struct{
Headers HeadersType
Payload PayloadType
}
func (*Broker) PublishAsync ¶
PublishAsync begins publishing one message and returns its completion result. Encoding and submission may apply backpressure before this method returns. Done closes after Sarama reports the final delivery outcome.
func (*Broker) PublishBatch ¶
PublishBatch submits every element of a slice to the given topic as one application-facing batch. Sarama determines the actual broker request boundaries from the configured batching limits. Broker controller should be connected - returns error otherwise. An empty slice is a successful no-op. Any non-slice value results in an error.
A partial failure is reported as a *PublishError holding one error per input element, which BatchErrors unpacks:
if err := broker.PublishBatch(ctx, topic, msgs); err != nil {
results, ok := BatchErrors(err)
if !ok {
// the call failed as a whole, nothing was published
}
for i, res := range results {
// res is nil when msgs[i] was published
}
}
func (*Broker) PublishBatchAsync ¶
PublishBatchAsync begins publishing every element of a slice and returns a completion result. All encodable messages are attempted even when another element fails. MessageErrors remains aligned with the original input order.
func (*Broker) Subscribe ¶
func (b *Broker) Subscribe( ctx context.Context, topic string, handler func(ctx context.Context, msg any, unmarshal func(any, any) error) error, ) (<-chan struct{}, error)
Subscribe subscribes to the given topic and processes incoming messages using the provided handler. The broker must be connected via Connect() before subscribing. Missing topics are never created by subscribing.
On success, returns a channel that is closed when message processing stops (e.g., due to context cancellation or Unsubscribe), and a nil error. On failure, returns a closed channel and an error.
The handler receives each message as an opaque `any` value, which is actually a pointer to the library's broker message structure (*BrokerMessage):
type BrokerMessage struct {
Headers map[string][]byte // raw header values as byte slices
Payload []byte // raw message payload as byte slice
}
A built-in unmarshal helper (FromBrokerMessage) is provided to simplify deserialization. Its target may be a pointer to any JSON-decodable value. A byte slice receives the raw payload. Envelope structs with exported Headers and Payload fields decode Kafka headers and payload separately.
Example envelope message type for unmarshaling:
type HeadersType struct {
CorrelationID string `json:"correlation_id"`
Token []byte `json:"token"`
}
type PayloadType struct {
UserID string `json:"user_id"`
Action string `json:"action"`
}
type Message struct {
Headers HeadersType
Payload PayloadType
}
Example handler using the built-in unmarshaler:
func handler(ctx context.Context, msg any, unmarshal func(any, any) error) error {
var m Message
if err := unmarshal(msg, &m); err != nil {
log.Printf("discard malformed message: %v", err)
return nil
}
// process m.Headers and m.Payload
return nil
}
Using FromBrokerMessage is optional. A handler may type-assert *BrokerMessage to inspect the raw broker message directly.
A nil handler result acknowledges the message. An error negatively acknowledges it; the same message is delivered again after NackBackoff and its partition does not advance. To stop retrying, the service must handle the terminal failure itself and return nil. Malformed or undecryptable data is normally terminal, while transient dependency errors are normally returned.
type BrokerMessage ¶
BrokerMessage is the Kafka library's transport representation. Application code normally uses Publish and FromBrokerMessage instead.
type Config ¶
type Config struct {
ControllerFactory ControllerFactory // Factory for creating Kafka controllers; used for testing.
SaramaFactory SaramaFactory // Factory for constructing Sarama clients; used for testing and customization.
DSN string `json:"dsn" yaml:"dsn" env:"KAFKA_DSN"`
// Consumer group ID, required for consumer. No defaults.
GroupID string `json:"group_id" yaml:"group_id" env:"KAFKA_GROUP_ID"`
// Default - ModeMixed.
Mode Mode `json:"mode" yaml:"mode" env:"KAFKA_MODE"`
// Broker addresses. When empty, DSN is used as the only address.
Hosts []string `json:"hosts" yaml:"hosts" env:"KAFKA_HOSTS" env-separator:","`
// Max message size (default - 10MB). Must be less than Sarama's
// maximum request size.
MaxBytes int `json:"max_bytes" yaml:"max_bytes" env:"KAFKA_MAX_BYTES"`
// How many times a missing topic is created before publishing fails. The
// default is 0, which disables automatic topic creation.
TopicCreateAttempts int `json:"topic_create_attempts" yaml:"topic_create_attempts" env:"KAFKA_TOPIC_CREATE_ATTEMPTS"`
// Delay between topic creation attempts, letting the cluster propagate
// the new topic metadata (default - 100ms).
TopicCreateBackoff time.Duration `json:"topic_create_backoff" yaml:"topic_create_backoff" env:"KAFKA_TOPIC_CREATE_BACKOFF"` //nolint:lll
// How many messages a producer buffers before sending them as one batch
// (default - 100).
BatchSize int `json:"batch_size" yaml:"batch_size" env:"KAFKA_BATCH_SIZE"`
// Max size of a batch in bytes (default - 1MB). Must be less than
// Sarama's maximum request size.
BatchBytes int64 `json:"batch_bytes" yaml:"batch_bytes" env:"KAFKA_BATCH_BYTES"`
// How long an incomplete batch waits before being flushed anyway
// (default - 10ms).
BatchTimeout time.Duration `json:"batch_timeout" yaml:"batch_timeout" env:"KAFKA_BATCH_TIMEOUT"`
// Delay before retrying a message whose handler returned an error
// (default - 1s).
NackBackoff time.Duration `json:"nack_backoff" yaml:"nack_backoff" env:"KAFKA_NACK_BACKOFF"`
// Offset auto commit (default - false).
AutoCommit bool `json:"auto_commit" yaml:"auto_commit" env:"KAFKA_AUTO_COMMIT"`
// Default - true.
ConnectionTest bool `json:"connection_test" yaml:"connection_test" env:"KAFKA_CONNECTION_TEST"`
}
Config is an internal structure for storing parameters.
type Controller ¶
type Controller interface {
Publish(ctx context.Context, topic string, msgs []BrokerMessage) ([]error, error)
Subscribe(ctx context.Context, topic string) (Subscription, error)
Close() error
}
Controller defines the interface for Kafka controller operations.
type ControllerAdapter ¶
type ControllerAdapter struct {
// contains filtered or unexported fields
}
ControllerAdapter implements Controller with Sarama-backed producer and consumer adapters.
func NewControllerAdapter ¶
func NewControllerAdapter( writer *ControllerWriter, consumerGroupFactory consumerGroupFactory, nackBackoff time.Duration, ) *ControllerAdapter
NewControllerAdapter creates a new adapter for Kafka controller.
func (*ControllerAdapter) Close ¶
func (ca *ControllerAdapter) Close() error
Close closes the producer adapter.
func (*ControllerAdapter) Publish ¶
func (ca *ControllerAdapter) Publish( ctx context.Context, topic string, msgs []BrokerMessage, ) ([]error, error)
Publish writes the messages through the asynchronous producer adapter.
func (*ControllerAdapter) Subscribe ¶
func (ca *ControllerAdapter) Subscribe(ctx context.Context, topic string) (Subscription, error)
Subscribe starts a Sarama consumer-group subscription.
type ControllerFactory ¶
type ControllerFactory interface {
New(cfg Config) (Controller, error)
}
ControllerFactory creates Kafka controllers.
type ControllerWriter ¶
type ControllerWriter struct {
// contains filtered or unexported fields
}
ControllerWriter adapts Sarama's asynchronous producer to synchronous and asynchronous per-message publish results.
func (*ControllerWriter) Close ¶
func (c *ControllerWriter) Close() error
Close flushes the asynchronous producer, waits for the result dispatcher, closes the topic manager, and is idempotent.
func (*ControllerWriter) Publish ¶
func (c *ControllerWriter) Publish( ctx context.Context, topic string, messages []BrokerMessage, ) ([]error, error)
Publish submits all messages to the asynchronous producer and blocks until every submitted message succeeds, fails, or the context is cancelled.
type Mode ¶
type Mode = string
Mode is a preferred usage mode: producer-only, consumer-only or full client (mixed). Either choice affects only the list of required options.
const ( // ModeProducer is used when the broker is only used to publish messages. // When this mode is set, consumer-specific options like GroupID are not required. ModeProducer Mode = "producer" // ModeConsumer is used when the broker is only used to subscribe to messages. // When this mode is set, GroupID option is required for consumer group management. ModeConsumer Mode = "consumer" // ModeMixed is used when the broker is used for both publishing and subscribing. // This is the default mode. When this mode is set, GroupID option is required // for any subscription operations. ModeMixed Mode = "mixed" )
type PublishError ¶
type PublishError struct {
Errs []error
}
PublishError reports a batch publish in which some messages were written and some were not. A batch that failed as a whole is reported as a plain error instead, since there is nothing to tell apart.
func (*PublishError) Error ¶
func (e *PublishError) Error() string
Error implements error interface.
func (*PublishError) Unwrap ¶
func (e *PublishError) Unwrap() []error
Unwrap exposes the per-message errors to errors.Is and errors.As.
type PublishResult ¶
type PublishResult interface {
Done() <-chan struct{}
Err() error
}
PublishResult reports completion of one asynchronous publication. Err must only be inspected after Done is closed.
type SaramaFactory ¶
type SaramaFactory interface {
NewClient(hosts []string, cfg *sarama.Config) (sarama.Client, error)
NewAsyncProducer(hosts []string, cfg *sarama.Config) (sarama.AsyncProducer, error)
NewClusterAdmin(hosts []string, cfg *sarama.Config) (sarama.ClusterAdmin, error)
NewConsumerGroup(hosts []string, groupID string, cfg *sarama.Config) (sarama.ConsumerGroup, error)
}
SaramaFactory constructs the Sarama clients used by the Kafka controller. Applications normally use the default implementation. The interface allows deterministic adapter tests and specialized client construction.
type Subscription ¶
type Subscription interface {
MessagesChannel() <-chan AcknowledgeableBrokerMessage
Cancel(ctx context.Context)
}
Subscription defines the interface for Kafka subscription operations.
type SubscriptionAdapter ¶
type SubscriptionAdapter struct {
// contains filtered or unexported fields
}
SubscriptionAdapter adapts a Sarama consumer group to Subscription.
func (*SubscriptionAdapter) Cancel ¶
func (s *SubscriptionAdapter) Cancel(ctx context.Context)
Cancel stops the consumer group and waits for bounded cleanup.
func (*SubscriptionAdapter) MessagesChannel ¶
func (s *SubscriptionAdapter) MessagesChannel() <-chan AcknowledgeableBrokerMessage
MessagesChannel returns messages delivered by the consumer group.