scout

README

Scout

Scout is a daemon for listening to a set of SNS topics and enqueuing anything it finds into sidekiq jobs. It's meant to extract processing of SQS from the rails apps that increasingly need to do so.

Usage

NAME:
   scout - SQS Listener
Poll SQS queues specified in a config and enqueue Sidekiq jobs with the queue items.
It gracefully stops when sent SIGTERM.

USAGE:
   scout [global options] command [command options] [arguments...]

VERSION:
   v1.6.0

COMMANDS:
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --config FILE, -c FILE       Load config from FILE, required
   --freq N, -f N               Poll SQS every N milliseconds (default: 100)
   --log-level value, -l value  Sets log level. Accepts one of: debug, info, warn, error
   --json, -j                   Log in json format
   --help, -h                   show help
   --version, -v                print the version

Configuration

The configuration requires 3 distinct sets of information. It needs information about how to connect to redis to enqueue jobs, credentials to talk to AWS and read SQS, and a mapping from SNS topics to sidekiq worker classes in the application. The structure looks like this.

redis:
  host: "localhost:9000"
  queue: "background"
  namespace: "test" # optional key
  password: "someoptionalpassword"  # optional key
aws:
  access_key: "super"
  secret_key: "secret"
  region: "us-best"
queue:
  name: "myapp_queue"
  topics:
    foo-topic: "FooWorker"
    bar-topic: "BazWorker"

None of this information is actually an example of anything other than the strucure of the file, so if you copy paste it you'll probably be disappointed.

Environment Variables

A few optional settings can also be configured by environment variable:

• SCOUT_SQS_MAX_NUMBER_OF_MESSAGES - Max number of SQS messages to fetch at once
• SCOUT_SQS_WAIT_TIME_SECONDS - Max seconds to wait for an SQS message per poll
• SCOUT_SQS_VISIBILITY_TIMEOUT - How long to hide an SQS message after receiving it

Versioning

Scout uses tagged commits that are compatible with go modules. The first module aware version of scout is version v1.5.0. We recommend that you also use go modules to guard against unexpected updates.

For legacy systems not using go modules, you can import using gopkg.in to pin to version 1. The import path is gopkg.in/enova/scout.v1.

Development

Scout uses go modules to manage it's dependencies, so you should clone it to a location outside your GOPATH. At that point all the standard toolchain commands do what they say on the box.

Testing

The normal test suite can be run as expected with go test. There are also two tagged files with expensive integration tests that require external services. They can be run as follows

 [FG-386] scout > go test -run=TestSQS -v -tags=sqsint
=== RUN   TestSQS_Init
--- PASS: TestSQS_Init (3.84s)
=== RUN   TestSQS_FetchDelete
--- PASS: TestSQS_FetchDelete (3.58s)
    PASS
ok     github.com/enova/scout  7.422s
 [FG-386] scout > go test -run=TestWorker -v -tags=redisint
=== RUN   TestWorker_Init
--- PASS: TestWorker_Init (0.00s)
=== RUN   TestWorker_Push
--- PASS: TestWorker_Push (0.00s)
PASS
ok      github.com/enova/scout  0.013s

The tests themselves (found in sqs_client_test.go and worker_client_test.go) explain what is required to run them. In particular, the SQS integration tests require that you provide AWS credentials to run them.

Documentation

Overview

Package main is entry point for scout