Kulay
An high speed message passing system between various queues and services.
Getting started
- Download the latest binary and add its location to your
PATH
- Create a config file, kulay.toml and copy the following contents to it.
[sqs]
[sqs.test_sg]
queue_url = "https://sqs.ap-southeast-1.amazonaws.com/12345678/test_sg"
region = "ap-southeast-1"
delete_msg = false
[sqs.test_us]
queue_url = "https://sqs.us-east-1.amazonaws.com/12345678/test_us"
region = "us-east-1"
delete_msg = false
[jsonl]
[jsonl.local_backup]
path = "/tmp/backup.jsonl"
rotate = true
batch = 100
[redisq]
[redisq.logbuffer]
host = "localhost"
port = "6379"
password = ""
database = 0
queue = "test"
- Point the environment variable
KULAY_CONF
to the above file's location
$ export KULAY_CONF=/path/to/kulay.toml
- Change the sqs url and redis host to your endpoints. With reference to the config file, you can route messages between various services
$ kulay -f sqs.test_sg -t redisq.logbuffer
$ kulay -f sqs.test_us -t sqs.test_sg
$ kulay -f sqs.test_sg -t jsonl.local_backup
$ kulay -f jsonl.local_backup -t redisq.logbuffer
Config file structure
Each section in the config file is a service which we can route messages to, It can be redis queue, redis pubsub, (AWS)SQS, rabbitmq, zeromq, csv or jsonl (json delimited). Section names are already predefined for a service.
Each sub section can be any of your favourite name strictly preceeded by the section name. The sub section will contain various options with respect to the service you are using. An example redis queue section
[redisq]
[redisq.logbuffer]
host = "localhost"
port = "6379"
password = ""
database = 0
queue = "test"
Here the logbuffer is a custom name you give to your subsection, it can be production_buffer or prodqueue or which ever makes more sense to you. The host
, port
, password
, database
are the options we need to initilize the redis client and queue
is the key name of the list in redis which will be used as queue.
Currently kulay supports passing messages between :
- Redis queue
- Redis pubsub
- AWS SQS
- Jsonl file read/write
Config structures for supported services :
Redis pubsub
[redispubsub]
[redispubsub.yourCustomName]
host = "localhost"
port = "6379"
password = "topsecret"
database = 0
channel = "mychannel"
host - Your redis hostname or IP address
port - Port in which redis runs, default is 6379
password - Password for your redis server, leave it as "" for no password
channel - The pubsub channel which you need to send or receive messages
Redis queue
[redisq]
[redisq.yourCustomName]
host = "localhost"
port = "6379"
password = "topsecret"
database = 0
queue = "mychannel"
host - Your redis hostname or IP address
port - Port in which redis runs, default is 6379
password - Password for your redis server, leave it as "" for no password
queue - The queue to which you will send or receive messages
SQS
[sqs]
[sqs.test_singapore]
queue_url = "https://sqs.ap-southeast-1.amazonaws.com/12345678/test_queue"
region = "ap-southeast-1"
delete_msg = true
queue_url - URL of the queue found in AWS console
region - The region where given queue was created
delete_msg - Delete flag, should be true if you want to delete the message from sqs after reading
Jsonl
jsonl]
[jsonl.local_backup]
path = "/tmp/backup.jsonl"
rotate = true
batch = 1000
path - Location where the files has to be created
rotate - If rotate flag is enabled, kulay will create a new file if the lines crosses the count in batch
field
batch - The line count for a single file if rotate=true
Built with
Versioning
This project uses SemVer for versioning. For the versions available, see the tags on this repository