sqs

package module
v0.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: MIT Imports: 23 Imported by: 0

README

messaging-sqs

SQS and SNS backends for messaging: the bus's third implementation, and the first outside NATS.

Go Reference Pipeline coverage

Part of the phpboyscout Go toolkit. This is a provider module of messaging, and its documentation lives with the core: messaging.go.phpboyscout.uk. Its decisions are spec 0007.

Two backends

FIFO Standard
what it is one FIFO topic per subject, the subject as the message group standard topics and queues, for throughput
delivery modes AtMostOnce, AtLeastOnce AtLeastOnce only: a standard queue may deliver a message after it has been deleted
ordering strict per subject, one message in flight per subject per queue best effort
deduplication on publish, by the event's identity none
survives a restart yes yes
lease the visibility timeout, extended in whole seconds the same
a declined message's delay the visibility timeout, rounded up to whole seconds, at most twelve hours from its receive, or from the call under MaxHeadroom() the visibility timeout, rounded up to whole seconds, at most twelve hours from its receive
patterns no: a subscription is a queue, and a queue is a name no

A subscription's Backoff and Delivery.NackAfter work on both: a declined message is hidden for its delay with ChangeMessageVisibility. SQS will not hide a message past twelve hours from the receive that took it, so a delay longer than what is left is cut to what is left, the message returns sooner than asked, and ConsumerStats.DelayShortened counts it (go/messaging spec 0013, D9). It is rare: a message that near the cap has been failing for most of a day.

Using all of FIFO's twelve hours

sqs.FIFO(settings, sqs.MaxHeadroom()) gives a FIFO subscription the headroom SQS was measured allowing: a decline waits its whole delay, a lease extends by its full AckWait on every heartbeat, and MaxProcessingTime may run past twelve hours. On 2026-10-12 a FIFO queue accepted a visibility of twelve hours from the call at any point in a receive, where a Standard queue refused anything past twelve hours from the receive (the measurement).

AWS documents the cap for both types, so this is opt-in. If SQS refuses a call as past the cap, the module retries it at what is left of twelve hours from the receive, which is what it does without the option: a decline cut that way counts in DelayShortened, and a lease with nothing left ends and counts in LeaseExpired.

Two backends rather than one with a flag, because a queue's type decides what the backend can honestly declare, and Capabilities is declared rather than discovered.

Operating it

Almost nothing is created. Topics, queues, their subscriptions and redrive policies are the operator's; the module resolves them by name at Subscribe and Publish and fails naming whichever is missing. A subscription's MaxDeliver must equal its queue's maxReceiveCount, because the receive count is the attempt and the redrive policy is where the budget is enforced.

The one exception is at-most-once fan-out. A subscription in that mode gets a queue of its own, created at Subscribe, subscribed to the subject's topic with raw delivery, and deleted at Retire. There is nothing for an operator to create, because it exists only for the life of one source. The access policy the module puts on it admits that topic and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "sns.amazonaws.com"},
    "Action": "sqs:SendMessage",
    "Resource": "<the queue's ARN>",
    "Condition": {"ArnEquals": {"aws:SourceArn": "<the topic's ARN>"}}
  }]
}
How many consumers a subject can have

About a hundred, at once. One FIFO topic sits in front of each subject, and AWS caps subscriptions per FIFO topic at 100. Every durable queue consuming a subject holds one, and so does every live at-most-once fan-out subscriber, so a fleet that scales those with its instance count reaches the ceiling at roughly a hundred instances per subject. It arrives at Subscribe, as SubscriptionLimitExceeded.

Two neighbouring quotas matter and are easy to miss. FIFO topics are capped at 1,000 per account, which one topic per subject makes the ceiling on subjects. Publish throughput is account-wide, counting Publish and PublishBatch across every topic of both varieties, at 3,000 messages a second for FIFO in most regions and 30,000 in us-east-1; many quiet subjects share that one budget, so throttling on publish is a likelier first symptom than any topic-count error.

All three are soft, and an increase is a support request. They were read on 2026-09-07 from AWS's SNS endpoints and quotas, which is not date-stamped and varies by region, so check it rather than trusting these numbers for capacity planning.

Choosing a wire format

Settings.Format writes every event this backend publishes. Unset, the writer is CloudEvents binary with the ce- prefix, which is what this module has always written. Each attribute is an SNS message attribute, and so is the content type, because neither service has a property for it. No CloudEvents binding covers SNS or SQS, so the prefix follows HTTP's.

Settings.Reads is the chain a received message is read with. Exactly one format must claim a message. Unset, the chain is binary and then structured, so a consumer can read structured before any producer writes it. To move a deployment to structured, upgrade the readers first and switch the writers after.

sqs.Standard(sqs.Settings{AWS: src, Format: cloudevents.Structured()})

Structured mode moves every attribute into the message body, with only content-type left as an SNS attribute. That buys two things binary cannot do:

  • More than ten attributes. SNS carries at most ten message attributes. In binary, an event's required attributes and its content type spend five of them, so an event with more than five extensions is refused at Publish. Structured carries it.
  • A payload that is not UTF-8. SNS takes a message only as UTF-8 text, and structured carries such a payload base64-encoded, as data_base64. Binary refuses it at Publish, because LocalStack 4.14 does not enforce the rule and AWS does.

It costs one thing. A subscription filter policy on an attribute moves to the body. In binary, a policy on ce-type, ce-source or an extension filters on message attributes. In structured, those values are in the body, so the policy has to be rewritten for FilterPolicyScope: MessageBody. A deployment that cannot change its policies stays on binary.

  • Attributes no format takes, such as one added by another producer, reach a handler through Delivery.Foreign(). They are never sent on.
  • A message no format can read never reaches a handler. That covers a message no format claims, one two formats both claim (the reader refuses to guess between them), and one a format claims but cannot decode. It's logged and counted in Unreadable. Under AtLeastOnce it is released at once, so it spends its receives quickly and the redrive policy moves it to the dead-letter queue. Under AtMostOnce it is deleted (spec 0015).

To exchange messages with .NET services, the MassTransit and NServiceBus formats work the same way, as a writer, in Reads beside the defaults, or both. The suite runs under each:

sqs.Standard(sqs.Settings{
	AWS:    src,
	Format: masstransit.MassTransit(),
	Reads:  []envelope.Format{cloudevents.Binary(cloudevents.HTTP), cloudevents.Structured(), masstransit.MassTransit()},
})

MassTransit puts the whole event in the body, so like structured it spends one SNS attribute, its content type. It refuses at publish an ID that is not a UUID and a payload that is not JSON. NServiceBus writes its own headers as SNS attributes, and they count toward the ten. Each format's rules are on its package: go/envelope-masstransit and go/envelope-nservicebus.

The CloudEvents formats are go/envelope-cloudevents's, the event is go/envelope's, and the decisions are spec 0014.

Testing

The unit gate needs nothing. The conformance rows run against LocalStack 4.14.0 in a container under ./test/integration, gated by INT_TEST_INTEGRATION=1 (just test-integration), in CI's Docker-in-Docker integration job. The suite runs once per format, TestFIFOStructured, TestStandardStructured, TestMassTransitFIFO and the rest beside the binary runs, each on its own topics and queues, which takes about fifty minutes. The interop tests are named format first because the suite derives durable names from the test's name, and the longest are at SQS's eighty characters.

Status

Pre-1.0, tracking the core. Both backends are built and green under conformance.RunDurable against LocalStack, Standard in the cheapest mode it declares, at-least-once.

Documentation

Overview

Package sqs implements the backend contract of gitlab.com/phpboyscout/go/messaging over Amazon SQS and SNS, twice.

FIFO is the ordered backend: one FIFO topic in front of each subject, the subject as the message group, deduplication on publish by the event's identity, and at-most-once by deletion before hand-off. Standard is the throughput backend: at-least-once only, because a standard queue may deliver a message even after it has been deleted. Both survive a restart, both keep a lease with the visibility timeout in whole seconds, and neither matches a pattern, because a subscription there is a queue and a queue is a name.

Almost nothing is created: topics, queues, subscriptions and redrive policies are the operator's, resolved by name, failing naming whichever is missing. The one exception is at-most-once fan-out, which creates its own queue at Subscribe, subscribes it to the subject's topic under a policy admitting only that topic, and deletes it at Retire, because the queue exists only for the life of one source.

A subject supports about a hundred consumers at once. AWS caps subscriptions per FIFO topic at 100, and every durable queue consuming the subject holds one, as does every live fan-out subscriber. The README carries that quota and its two neighbours.

This module leans on gitlab.com/phpboyscout/go/awsclient for configuration and credentials and writes the receiving, publishing and settling against aws-sdk-go-v2's sqs and sns services, which appear nowhere else in the estate.

Settings.Format writes what the backend publishes, CloudEvents binary with the ce- prefix as SNS attributes when unset, and Settings.Reads is the chain it reads with, binary then structured when unset. Structured carries what binary cannot, more than ten attributes and a payload that is not UTF-8, at the cost of moving any filter policy on an attribute to the message body (messaging spec 0014).

Specified by https://gitlab.com/phpboyscout/go/messaging/-/wikis/specs/0007-the-sqs-backend

Index

Constants

This section is empty.

Variables

View Source
var ErrClosed = errors.NewSentinel("messaging_sqs.closed", "messaging-sqs: the session is closed")

ErrClosed is returned by Publish and Subscribe after Close: the message went nowhere, or the source would outlive its session, and a caller must know.

View Source
var ErrNoQueue = errors.NewSentinel("messaging_sqs.no_queue",
	"messaging-sqs: the queue this subscription consumes does not exist; a bus resolves a queue and never creates one")

ErrNoQueue is returned by Subscribe when the queue a subscription names does not exist. The bus binds, and never creates (0005 D9, 0007 D2).

View Source
var ErrNoTopic = errors.NewSentinel("messaging_sqs.no_topic",
	"messaging-sqs: the topic in front of this subject does not exist; a bus resolves a topic and never creates one")

ErrNoTopic is returned by Subscribe and Publish when the topic in front of a subject does not exist.

Functions

func FIFO

func FIFO(s Settings, opts ...FIFOOption) messaging.Backend

FIFO is the ordered backend (0007 D1): one FIFO topic per subject with the subject as the message group, deduplication on publish by the event's identity, both delivery modes, and one message in flight per subject per queue, which is what strict ordering costs on SQS.

func Standard added in v0.2.0

func Standard(s Settings) messaging.Backend

Standard is the throughput backend (0007 D1): standard topics and queues, at-least-once only, because a standard queue may deliver a message even after it has been deleted, which is exactly the case at-most-once forbids. The suite holds it to the general rows in that mode (0005 D12, the cheapest declared mode).

Types

type ByName

type ByName struct{}

ByName is the default Topology: the derived names are the operator's names.

func (ByName) QueueURL

func (ByName) QueueURL(ctx context.Context, c Clients, name string, _ messaging.SourceSpec) (string, error)

QueueURL resolves the queue by name.

func (ByName) TopicARN

func (ByName) TopicARN(ctx context.Context, c Clients, name string, _ string) (string, error)

TopicARN resolves the topic by name among the account's topics. SNS has no lookup by name, so this pages ListTopics once per subject; a deployment with many topics should supply a Topology that knows its ARNs.

type Clients

type Clients struct {
	SQS *awssqs.Client
	SNS *awssns.Client
}

Clients are the two service clients a session holds, handed to a Topology.

type FIFOOption added in v0.5.0

type FIFOOption func(*backend)

FIFOOption configures the FIFO backend.

func MaxHeadroom added in v0.5.0

func MaxHeadroom() FIFOOption

MaxHeadroom lets a FIFO message stay hidden past twelve hours from its receive: a decline waits its whole delay, a lease extends by its full AckWait, and MaxProcessingTime may exceed twelve hours. SQS was measured enforcing only its per-call twelve hours on FIFO, against its own guide, so a refusal falls back to the default cut (messaging 0013 D10).

type Settings

type Settings struct {
	// AWS resolves the configuration the SQS and SNS clients are built from.
	// Required.
	AWS awsclient.Source

	// Topology resolves the names this module derives into what the operator
	// created. Nil means ByName: a queue by GetQueueUrl on its derived name,
	// a topic by its derived name among the account's topics (0007 D2, D3).
	Topology Topology

	// Logger receives what the backend cannot return: a message on a queue no
	// format could read, a heartbeat that failed. Nil discards.
	Logger *slog.Logger
	// Format writes every event this backend publishes. Unset, it is
	// CloudEvents binary with the ce- prefix (0014 D4, D6). Structured moves
	// every attribute into the body, which lifts SNS's ten-attribute cap and
	// moves any filter policy on them to MessageBody scope.
	Format envelope.Format

	// Reads is the chain a received message is read with, strictly: one
	// format must claim it. Unset, it is binary with the ce- prefix, then
	// structured.
	Reads []envelope.Format
}

Settings is what both backends are built from.

type Topology

type Topology interface {
	// QueueURL is the queue a subscription consumes. The spec is the
	// subscription's, so an operator's tool can see what it asks for.
	QueueURL(ctx context.Context, clients Clients, name string, spec messaging.SourceSpec) (string, error)

	// TopicARN is the topic in front of a subject.
	TopicARN(ctx context.Context, clients Clients, name string, subject string) (string, error)
}

Topology is the operator's answer to the names the module derives (0007 D3). The module creates nothing; a Topology that does is the operator's tool, and a test fixture is one.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL