pubsub/

directory
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 20, 2018 License: Apache-2.0

README

Example on how to use AWS SNS + SQS for pub/sub communication

We're going to set up two services to follow the pub/sub pattern using AWS SNS and AWS SQS. One service is api-sns-pub and the other service is sqs-sub. They can both be found at https://github.com/NYTimes/gizmo/tree/master/examples/pubsub. For more information about pub/sub pattern, SNS, and SQS check out http://www.infoq.com/articles/AmazonPubSub.

High level overview
  1. Create SNS Topic
  2. Create SQS Queue
  3. Get the SQS Queue to subscribe to the SNS Topic
  4. Set up api-sns-pub example service to send messages to the SNS Topic
  5. Set up sqs-sub example service to listen to the SQS Queue
Set up a topic with AWS SNS console
  1. Login to your AWS account
  2. Go to https://console.aws.amazon.com/sns
  3. Click on "Topics" on the left hand side
  4. Click "Create new topic"
  5. Topic Name = TestTopic (or whatever you want)
  6. TestTopic = TestTopic (or whatever you want)
Set up an AWS SQS Queue with AWS console
  1. Go to https://console.aws.amazon.com/sqs
  • Click on Create New Queue
  • Queue Name = TestQueue (or whatever you want)
  • Click on "TestQueue" and then click "Queue Actions" dropdown and chooose "Subscribe Queue to SNS Topic"
  • Choose the topic we created earlier (TestTopic) from the correct region and then hit "Subscribe"
Change SNS Subscription to Raw Message Delivery
  1. Go to https://console.aws.amazon.com/sns
  • Click on the blue ARN text next to "TestTopic" to take you to the "Topic Details: TestTopic" page
  • Click on the checkmark besides the ARN Subscription ID, which is from your SQS "TestQueue"
  • Click "Edit Subscription Attributes" from the dropdown menu titled "Other Subscription Actions"
  • Turn on "Raw Message Delivery"
Set up logging files
  1. gizmo logs to the directory specified in config.json. The example project is pointing to /var/nyt/logs/cats-publisher/ for the api-sns-pub service and /var/nyt/logs/cats-subscriber/ for the sqs-sub service
  • You must create this directory and add access.log and app.log inside it.
  • You might not have permission to run the following commands in which case you'll have to put sudo before them.
  • In your terminal run mkdir -p /var/nyt/logs/
  • run cd /var/nyt/logs/
  • run mkdir cats-publisher
  • run mkdir cats-subscriber
  • run touch cats-publisher/access.log
  • run touch cats-publisher/app.log
  • run touch cats-subscriber/app.log
  • To allow the api-sns-pub server to access these files run the following commands:
  • chmod -R 777 cats-publisher/
  • chmod -R 777 cats-subscriber/
Set up api-sns-pub example server
  1. run go get github.com/NYTimes/gizmo inside your $GOPATH folder
  • run cd $GOPATH/src/github.com/NYimes/gizmo/examples/pubsub/api-sns-pub
  • run go get ./...
  • update the config.json file to point to your newly created ARN for "Topic". Example: arn:aws:sns:us-east-1:123456789:TestTopic
  • also update the config.json file to contain your AccessKey and SecretKey
  • ensure that the region in your config.json file is the same as the region you created your SNS Topic and SQS Queue
  • run go install ./...
  • run api-sns-pub (this should be in your $GOPATH/bin folder)
  • Your api-sns-pub server should now be running!
Push a message to your topic
  1. Call the server using a PUT request to localhost:8080/svc/nyt/cats with a body like the following json:
{
	"url":"http://www.nytiems.com/cats-article",
	"title":"cats cats cats"
}
  1. CURL example: curl -X PUT -H "Content-Type: application/json" -d '{"url":"http://www.nytiems.com/cats-article","title":"cats cats cats"}' localhost:8080/svc/nyt/cats
  2. You should see {"status":"success!"}
  3. If you check your SQS Queue, you should now have one message in it
Read a message from your SQS Queue
  1. run cd $GOPATH/src/github.com/NYTimes/gizmo/examples/pubsub/sqs-sub
  • run go get ./...
  • update the config.json file to contain your AccessKey and SecretKey
  • also make sure it points to the correct region
  • run go install ./...
  • run sqs-sub
  • Your sqs-sub server should now be running!
  • You should have seen the message(s) you sent earlier that were stacked on the SQS Queue
  • Now, when you run your PUT /svc/nyt/cats request, it will update the SNS, which SQS is subscribed to, and your sqs-sub service is polling the SQS Queue. If there is a message on the queue, it will be deleted from the queue and read by sqs-sub.

Congrats on setting up pub/sub communication with AWS SNS + SQS!


Troubleshooting
  • Make sure the region your created your SNS topic and SQS Queue matches the region in your config.json files.
  • Make sure the Access Key and Secret Access Key you passed into the config.json have appropriate credentials for communicating with SNS and SQS

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL