goreactor

README

Goreactor, trigger a message and execute commands

The current version just support inputs from AWS SQS, in next versions will support also HTTP/S and Redis QUEUES.

Usage case, message from SQS

In the next example, we will run a command after receiving a message from SQS, the message received will look like the JSON below

Message received from SQS

{
    "Progress": 50,
    "AccountId": "9999999999",
    "Description": "Launching EC2 instance: i-00000009999",
    "RequestId": "5f88cdf3-311c-4d61-bdec-0000000000",
    "EndTime": "2018-04-18T14:17:22.779Z",
    "AutoScalingGroupARN": "arn:aws:autoscaling:eu-west-1:9999999999:autoScalingGroup:0b35d38a-8270-45d0-a0d8-0000000000:autoScalingGroupName/SOMEGROUPNAME",
    "ActivityId": "5f88cdf3-311c-4d61-bdec-0000000000",
    "StartTime": "2018-04-18T14:16:26.804Z",
    "Service": "AWS Auto Scaling",
    "Time": "2018-04-18T14:17:22.779Z",
    "EC2InstanceId": "i-00000009999", // <--- We read this
    "StatusCode": "InProgress",
    "StatusMessage": "",
    "Details": {
        "Subnet ID": "subnet-XXXXXXXX",
        "Availability Zone": "eu-west-1a"
    },
    "AutoScalingGroupName": "SOMEGROUPNAME", // <--- We read this
    "Cause": "At 2018-04-18T14:16:21Z a user request update of AutoScalingGroup constraints to min: 1, max: 5, desired: 3 changing the desired capacity from 2 to 3....",
    "Event": "autoscaling:EC2_INSTANCE_LAUNCH" // <--- We read this, as condition
}

Configuration in go reactor

In the goreactor, we are ready to listen to messages and execute the command.

Please put attention in the "cond" parameter (is a condition, like an "if"). This means that the command will be executed JUST if we can find "Event = autoscaling:EC2_INSTANCE_LAUNCH" in the JSON. If this key is not there or the value is not what we defined, then goreactor will ignore the message

[logstream]
logstream = "stdout"

[[reactor]]
concurrent = 10
delay = "5s"
keepAliveInterval = "30s"
input = "sqs"
url = "https://sqs.eu-west-1.amazonaws.com/9999999999/testing"
profile = "default"
region = "eu-west-1"
maxNumberOfMessages = 10
output = "cmd"
cond = [
    { "$.Event" = "autoscaling:EC2_INSTANCE_LAUNCH" }
]
cmd = "/usr/local/bin/do-something-with-the-instance"
args = ["asg=$.AutoScalingGroupName", "instance_id=$.EC2InstanceId"]
workingDirectory = "/path/to/process/wd/"

The daemon will execute a command like

/usr/local/bin/do-something-with-the instance asg=SOMEGROUPNAME instance_id=i-00000009999

Arguments of a reactor

• concurrent - Maximum of commands (cmd) that can run concurrently
• delay - Period of time that a new command runs without concurrency, after that period the next message will start. Format https://pkg.go.dev/time#ParseDuration
• keepAliveInterval - A signal will be sent to the underline input source to inform that the process is still running. Format https://pkg.go.dev/time#ParseDuration. In SQS input will trigger a request ChangeMessageVisibility (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html)

Arguments of a reactor SQS

• url - AWS SQS URL
• region - AWS Region
• profile - AWS Profile
• maxnumberofmessages - AWS SQS setting: The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned). Valid values: 1 to 10. Default: 1.
• noblocking is a boolean value that defines the parallelims of the reactors
  • false (default): send the message to the reactors in sequence
  • true: send the message in parallel to avoid blocking

Set working directory

The working directory of the process can be set with workingDirectory.

Execute as a specific user

It may be desired to execute some commands as a specific user.

NOTE: that this only works in unix or unix like systems and have only been tested in linux. In other OS, this will be ignored.

In order to do so, each [[reactor]] entry may have a user value defined with the desired user. The env will be empty (only containing the HOME - which can be overridden) unless you specify it in a env section that will only be used if user is defined.

[[reactor]]
# (...) All the desired values
user = "john"
env = [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"AWS_PROFILE=stage"
]

This would run the reactor using john user and 2 environ: HOME set to the home defined for the john user, PATH defined to "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" and AWS_PROFILE set to "stage"

DO NOT USE SET THE SETUID BIT FOR GOREACTOR!!!!

Substitutions in args entry in the config file

• $.path.with[0].jq.format

  jq like format after $. See Usage case, message from SQS for an example

• $.path.to.array...

  If the jq like expression (Starting with $.) is the only text in the argument and ends with ... See examples/ARRAY.md for an example

  will expand the array in the specified path in different arguments.

  The use of this will replace the argument with N arguments where N is the number of elements in the array.

  NOTE: Should be the only text in the given argument element on the array.

• ${variableName}

  Variables will be substituted in the arguments if the variable is in the following list. (This can also come from an expanded array)

  NOTE: Variables are case sensitive and curly brackets are mandatory

  Currently, we have the following variables:

  • CreationTimestampMilliseconds is the message creation time with in milliseconds
  • CreationTimestampSeconds is the message creation time in seconds See examples/ARRAY.md for an example

• $..

  If you want to select the full message you should use $.. instead of $..

• $.Message

  If the message has a field "Message", the content of the original message will be replaced with the content of the field $.Message.

• Indented json

  The library used to parse jq like expressions sometimes has problems with new lines. It is better to use json messages without new lines or other indentations.

Log outputs to stdout or firehose

You can set logging to stdout with

[logstream]
logstream = "stdout"

You can set logging to firehose with

[logstream]
logstream = "firehose"
streamname = "example-goreactor-logs"
region = "eu-west-1"

You can disable logging if you leave logstream empty or set to none. Or not defining logstream at all

[logstream]
logstream = ""

[logstream]
logstream = "none"

The output will be a json per output line with the following format:

{"Host":"RUNNER_HOSTNAME","Pid":44274,"RID":1,"TID":1,"Line":0,"Output":"./print_some_lines_and_exit ","Status":"CMD","Timestamp":1635149955}
{"Host":"RUNNER_HOSTNAME","Pid":44274,"RID":1,"TID":1,"Line":1,"Output":"first line","Status":"RUN","Timestamp":1635149955}
{"Host":"RUNNER_HOSTNAME","Pid":44274,"RID":1,"TID":1,"Line":2,"Output":"second line","Status":"RUN","Timestamp":1635149955}
{"Host":"RUNNER_HOSTNAME","Pid":44274,"RID":1,"TID":1,"Line":3,"Output":"last line","Status":"RUN","Timestamp":1635149955}
{"Host":"RUNNER_HOSTNAME","Pid":44274,"RID":1,"TID":1,"Line":4,"Status":"END","Elapse":3.067383247,"Timestamp":1635149955}