Version: v1.3.0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: May 21, 2023 License: MIT


license Go Report Card Go Reference Artifact Hub Docker Pulls


sqs-to-sns is an utility written in Go to forward messages from AWS SQS Queues to AWS SNS Topics.

Created by gh-md-toc

Build and run

git clone https://github.com/udhos/sqs-to-sns

cd sqs-to-sns

./build.sh ;# compile

sts-to-sns ;# run the executable


Env vars

export QUEUES=queues.yaml ;# queue list configuration file

# Health check
# HINT: Recommended for livenessProbe only, not for readinessProbe

export HEALTH_ADDR=:8888
export HEALTH_PATH=/health

# Prometheus metrics

export METRICS_ADDR=:3000
export METRICS_PATH=/metrics
export METRICS_NAMESPACE=sqstosns

# These env vars define global defaults for per-queue config

export QUEUE_ROLE_ARN=arn:aws:iam::111111111111:role/sqs_consumer
export TOPIC_ROLE_ARN=arn:aws:iam::222222222222:role/sns_producer

export READERS=1                  ;# number of goroutines reading from SQS queue
export WRITERS=10                 ;# number of goroutines writing to SNS topic (recommended: 10 * READERS)
export BUFFER=20                  ;# buffer size between readers and writers (recommended: 20 * READERS)
export READ_ERROR_COOLDOWN=10s    ;# cooldown holdtime after read errors
export WRITE_ERROR_COOLDOWN=10s   ;# cooldown holdtime after write errors
export DELETE_ERROR_COOLDOWN=10s  ;# cooldown holdtime after delete errors
export EMPTY_RECEIVE_COOLDOWN=10s ;# cooldown holdtime after empty receive
export COPY_ATTRIBUTES=true       ;# enable copying of message attributes from SQS message to SNS message
export DEBUG=true                 ;# enable debug logs

Queue list configuration file

$ cat queues.yaml
- id: q1
  # required
  queue_url: https://sqs.us-east-1.amazonaws.com/111111111111/queue_name1
  topic_arn: arn:aws:sns:us-east-1:222222222222:topic_name1
  # optional
  #queue_role_arn: arn:aws:iam::111111111111:role/sqs_consumer1
  #topic_role_arn: arn:aws:iam::222222222222:role/sns_producer1
  #readers: 1
  #writers: 10 # recommended: 10*readers
  #buffer: 20  # recommended: 20*readers
  #error_cooldown_read: 10s
  #error_cooldown_write: 10s
  #error_cooldown_delete: 10s
  #empty_receive_cooldown: 10s
  #copy_attributes: true
  #debug: true
- id: q2
  # required
  queue_url: https://sqs.us-east-1.amazonaws.com/111111111111/queue_name2
  topic_arn: arn:aws:sns:us-east-1:222222222222:topic_name2
  # optional
  #queue_role_arn: arn:aws:iam::111111111111:role/sqs_consumer2
  #topic_role_arn: arn:aws:iam::222222222222:role/sns_producer2
  #readers: 1
  #writers: 10 # recommended: 10*readers
  #buffer: 20  # recommended: 20*readers
  #error_cooldown_read: 10s
  #error_cooldown_write: 10s
  #error_cooldown_delete: 10s
  #empty_receive_cooldown: 10s
  #copy_attributes: true
  #debug: true


You can use the "queue role ARN" to specify a role to access the source queue, and "topic role ARN" to specify a role to access the destination topic.

The role in "queue role ARN" must allow actions sqs:ReceiveMessage and sqs:DeleteMessage to source queue.

The role in "topic role ARN" must allow action sns:Publish to destination topic.

Prometheus Metrics

# HELP sqstosns_buffer How many SQS messages are buffered with us, partitioned by queue.
# TYPE sqstosns_buffer gauge

# HELP sqstosns_receive_total How many SQS receives called, partitioned by queue.
# TYPE sqstosns_receive_total counter

# HELP sqstosns_receive_error_total How many SQS receives errored, partitioned by queue.
# TYPE sqstosns_receive_error_total counter

# HELP sqstosns_receive_empty_total How many SQS empty receives, partitioned by queue.
# TYPE sqstosns_receive_empty_total counter

# HELP sqstosns_receive_messages_total How many SQS messages received, partitioned by queue.
# TYPE sqstosns_receive_messages_total counter

# HELP sqstosns_publish_error_total How many SNS publishes errored, partitioned by queue.
# TYPE sqstosns_publish_error_total counter

# HELP sqstosns_delete_error_total How many SQS deletes errored, partitioned by queue.
# TYPE sqstosns_delete_error_total counter

# HELP sqstosns_delivery_total How many SQS deliveries fully processed, partitioned by queue.
# TYPE sqstosns_delivery_total counter

# HELP sqstosns_delivery_duration_seconds How long it took to fully process the delivery, partitioned by queue.
# TYPE sqstosns_delivery_duration_seconds histogram

Utility to populate SQS queue

Use batch-sqs to send messages to an SQS queue.

CAUTION: With default parameters you would inject your SQS with 10000 messages at a rate of ~1000 messages/sec.

batch-sqs -queueURL https://sqs.us-east-1.amazonaws.com/111111111111/queue_name

2023/04/17 00:27:04 batch-sqs: sent=10000 interval=9.275251836s rate=1111.111111111111 messages/sec


  • Forward messages.
  • Docker image.
  • Message attributes.
  • Helm chart.
  • Multiple queues.
  • Metrics.
  • Health check.
  • Tests.


Docker hub:


Run from docker hub:

docker run -p 8080:8080 --rm udhos/sqs-to-sns:0.0.0

Build recipe:


docker push udhos/sqs-to-sns:0.0.0

Helm chart

Using the repository

See https://udhos.github.io/sqs-to-sns/.


mkdir charts
cd charts
helm create sqs-to-sns

Then edit files.


helm lint ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml

Test rendering chart templates locally

helm template sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml

Render templates at server

helm install sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml --dry-run

Generate files for a chart repository

A chart repository is an HTTP server that houses one or more packaged charts. A chart repository is an HTTP server that houses an index.yaml file and optionally (*) some packaged charts.

(*) Optionally since the package charts could be hosted elsewhere and referenced by the index.yaml file.

├── index.yaml
└── sqs-to-sns-0.1.0.tgz

See script update-charts.sh:

# generate chart package from source
helm package ./charts/sqs-to-sns -d ./docs

# regenerate the index from existing chart packages
helm repo index ./docs --url https://udhos.github.io/sqs-to-sns/


helm install sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml


helm upgrade sqs-to-sns ./charts/sqs-to-sns --values charts/sqs-to-sns/values.yaml --install


helm uninstall sqs-to-sns


Path Synopsis
Package main implements an utility.
Package main implements an utility.
Package main implements the program.
Package main implements the program.
Package sqsclient provides sqs utilities.
Package sqsclient provides sqs utilities.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL