kubemq-targets

README

KubeMQ Targets

KubeMQ Targets connects KubeMQ Message Broker with external systems and cloud services.

KubeMQ Targets allows us to build a message-based microservices architecture on Kubernetes with minimal efforts and without developing connectivity interfaces between KubeMQ Message Broker and external systems such as databases, cache, messaging, and REST-base APIs.

Key Features:

• Runs anywhere - Kubernetes, Cloud, on-prem, anywhere
• Stand-alone - small docker container / binary
• Single Interface - One interface all the services
• Any Service - Support all major services types (databases, cache, messaging, serverless, HTTP, etc.)
• Plug-in Architecture Easy to extend, easy to connect
• Middleware Supports - Logs, Metrics, Retries, and Rate Limiters
• Easy Configuration - simple yaml file builds your topology

Concepts

KubeMQ Targets building blocks are:

• Binding
• Source
• Target
• Request/Response

Binding

Binding is a 1:1 connection between Source and Target. Every Binding runs independently.

Target

Target is an external service that exposes an API allowing to interact and serve his functionalists with other services.

Targets can be Cache systems such as Redis and Memcached, SQL Databases such as Postgres and MySql, and even an HTTP generic Rest interface.

KubeMQ Targets integrate each one of the supported targets and service requests based on the request data.

A list of supported targets is below.

Standalone Services

Category	Target	Kind	Configuration	Example
Cache
	Redis	cache.redis	Usage	Example
	Memcached	cache.memcached	Usage	Example
	Hazelcast	cache.hazelcast	Usage	Example
Stores/db
	Postgres	stores.postgres	Usage	Example
	Mysql	stores.mysql	Usage	Example
	MSSql	stores.mssql	Usage	Example
	MongoDB	stores.mongodb	Usage	Example
	Elastic Search	stores.elastic-search	Usage	Example
	Cassandra	stores.cassandra	Usage	Example
	Couchbase	stores.couchbase	Usage	Example
	Percona	stores.percona	Usage	Example
	Cockroachdb	stores.cockroachdb	Usage	Example
	Aerospike	stores.aerospike	Usage	Example
	RethinkDB	stores.rethinkdb	Usage	Example
	SingleStore	stores.singlestore	Usage	Example
	Crate.io	stores.crate	Usage	Similar to postgres
Messaging
	Kafka	messaging.kafka	Usage	Example
	Nats	messaging.nats	Usage	Example
	RabbitMQ	messaging.rabbitmq	Usage	Example
	MQTT	messaging.mqtt	Usage	Example
	ActiveMQ	messaging.activemq	Usage	Example
	IBM-MQ	messaging.ibmmq	Usage	Example
Storage
	Minio/S3	storage.minio	Usage	Example
	hadoop/hdfs	storage.hdfs	Usage	Example
	Filesystem	storage.filesystem	Usage
Serverless
	OpenFaas	serverless.openfaas	Usage	Example
Http
	Http	http	Usage	Example
Testing
	Echo	echo	Usage

Google Cloud Platform (GCP)

Category	Target	Kind	Configuration	Example
Cache
	Redis	gcp.cache.redis	Usage	Example
	Memcached	gcp.cache.memcached	Usage	Example
Stores/db
	Postgres	gcp.stores.postgres	Usage	Example
	Mysql	gcp.stores.mysql	Usage	Example
	BigQuery	gcp.bigquery	Usage	Example
	BigTable	gcp.bigtable	Usage	Example
	Firestore	gcp.firestore	Usage	Example
	Spanner	gcp.spanner	Usage	Example
	Firebase	gcp.firebase	Usage	Example
Messaging
	Pub/Sub	gcp.pubsub	Usage	Example
Storage
	Storage	gcp.storage	Usage	Example
Serverless
	Functions	gcp.cloudfunctions	Usage	Example

Amazon Web Service (AWS)

Category	Target	Kind	Configuration	Example
Stores/db
	Athena	aws.athena	Usage	Example
	DynamoDB	aws.dynamodb	Usage	Example
	Elasticsearch	aws.elasticsearch	Usage	Example
	KeySpaces	aws.keyspaces	Usage	Example
	MariaDB	aws.rds.mariadb	Usage	Example
	MSSql	aws.rds.mssql	Usage	Example
	MySQL	aws.rds.mysql	Usage	Example
	Postgres	aws.rds.postgres	Usage	Example
	RedShift	aws.rds.redshift	Usage	Example
	RedShiftSVC	aws.rds.redshift.service	Usage	Example
Messaging
	AmazonMQ	aws.amazonmq	Usage	Example
	msk	aws.msk	Usage	Example
	Kinesis	aws.kinesis	Usage	Example
	SQS	aws.sqs	Usage	Example
	SNS	aws.sns	Usage	Example
Storage
	s3	aws.s3	Usage	Example
Serverless
	lambda	aws.lambda	Usage	Example
Other
	Cloud Watch Logs	aws.cloudwatch.logs	Usage	Example
	Cloud Watch Events	aws.cloudwatch.events	Usage	Example
	Cloud Watch Metrics	aws.cloudwatch.metrics	Usage	Example

Microsoft Azure

Category	Target	Kind	Configuration	Example
Stores/db
	Azuresql	azure.stores.azuresql	Usage	Example
	Mysql	azure.stores.mysql	Usage	Example
	Postgres	azure.stores.postgres	Usage	Example
Storage
	Blob	azure.storage.blob	Usage	Example
	Files	azure.storage.files	Usage	Example
	Queue	azure.storage.queue	Usage	Example
EventHubs
	EventHubs	azure.eventhubs	Usage	Example
ServiceBus
	ServiceBus	azure.servicebus	Usage	Example

Source

The source is a KubeMQ connection (in subscription mode), which listens to requests from services and route them to the appropriate target for action, and return back a response if needed.

KubeMQ Targets supports all of KubeMQ's messaging patterns: Queue, Events, Events-Store, Command, and Query.

Type	Kind	Configuration
Queue	kubemq.queue	Usage
Queue-Stream	kubemq.queue-stream	Usage
Events	kubemq.events	Usage
Events Store	kubemq.events-store	Usage
Command	kubemq.command	Usage
Query	kubemq.query	Usage

Request / Response

Request

A request is an object that sends to a designated target with metadata and data fields, which contains the needed information to perform the requested data.

Request Object Structure

Field	Type	Description
metadata	string, string object	contains metadata information for action
data	bytes array	contains raw data for action

Example

Request to get a data from Redis cache for the key "log"

{
  "metadata": {
    "method": "get",
    "key": "log"
  },
  "data": null
}

Response

The response is an object that sends back as a result of executing an action in the target

Response Object Structure

Field	Type	Description
metadata	string, string object	contains metadata information result for action
data	bytes array	contains raw data result
is_error	bool	indicate if the action ended with an error
error	string	contains error information if any

Example

Response received on request to get the data stored in Redis for key "log"

{
  "metadata": {
    "result": "ok",
    "key": "log"
  },
  "data": "SU5TRVJUIElOVE8gcG9zdChJRCxUSVRMRSxDT05URU5UKSBWQUxVRVMKCSAgICAgICAgICAgICAgICAgICAgICA"
}

Installation

Kubernetes

1. Install KubeMQ Cluster

kubectl apply -f https://get.kubemq.io/deploy

1. Run Redis Cluster deployment yaml

kubectl apply -f https://raw.githubusercontent.com/kubemq-hub/kubemq-targets/master/redis-example.yaml

1. Run KubeMQ Targets deployment yaml

kubectl apply -f https://raw.githubusercontent.com/kubemq-hub/kubemq-targets/master/deploy-example.yaml

Binary (Cross-platform)

Download the appropriate version for your platform from KubeMQ Targets Releases. Once downloaded, the binary can be run from anywhere.

Ideally, you should install it somewhere in your PATH for easy use. /usr/local/bin is the most probable location.

Running KubeMQ Targets

./kubemq-targets --config config.yaml

Windows Service

1. Download the Windows version from KubeMQ Targets Releases. Once downloaded, the binary can be installed from anywhere.
2. Create config.yaml configuration file and save it to the same location of the Windows binary.

Service Installation

Run:

kubemq-targets.exe --service install

Service Installation With Username and Password

Run:

kubemq-targets.exe --service install --username {your-username} --password {your-password}

Service UnInstall

Run:

kubemq-sources.exe --service uninstall

Service Start

Run:

kubemq-targets.exe --service start

Service Stop

Run:

kubemq-targets.exe --service stop

Service Restart

Run:

kubemq-targets.exe --service restart

NOTE: When running under Windows service, all logs will be emitted to Windows Events Logs.

Configuration

Build Wizard

KubeMQ Targets configuration can be build with Build and Deploy tool https://build.kubemq.io/#/sources

Structure

Config file structure:

apiPort: 8080 # kubemq targets api and health end-point port
bindings:
  - name: clusters-sources # unique binding name
    properties: # Bindings properties such middleware configurations
      log_level: error
      retry_attempts: 3
      retry_delay_milliseconds: 1000
      retry_max_jitter_milliseconds: 100
      retry_delay_type: "back-off"
      rate_per_second: 100
    source:
      kind: kubemq.query # source kind
      name: name-of-sources # source name 
      properties: # a set of key/value settings per each source kind
        .....
    target:
      kind:cache.redis # target kind
      name: name-of-target # targets name
      properties: # a set of key/value settings per each target kind
        - .....

Build Wizard

KubeMQ Targets configuration can be build with --build flag

./kubemq-targets --build

Properties

In bindings configuration, KubeMQ targets support properties setting for each pair of source and target bindings.

These properties contain middleware information settings as follows:

Logs Middleware

KubeMQ targets support level based logging to console according to as follows:

Property	Description	Possible Values
log_level	log level setting	"debug","info","error"
		"" - indicate no logging on this bindings

An example for only error level log to console:

bindings:
  - name: sample-binding 
    properties: 
      log_level: error
    source:
    ......  

Retry Middleware

KubeMQ targets support Retries' target execution before reporting of error back to the source on failed execution.

Retry middleware settings values:

Property	Description	Possible Values
retry_attempts	how many retries before giving up on target execution	default - 1, or any int number
retry_delay_milliseconds	how long to wait between retries in milliseconds	default - 100ms or any int number
retry_max_jitter_milliseconds	max delay jitter between retries	default - 100ms or any int number
retry_delay_type	type of retry delay	"back-off" - delay increase on each attempt
		"fixed" - fixed time delay
		"random" - random time delay

An example for 3 retries with back-off strategy:

bindings:
  - name: sample-binding 
    properties: 
      retry_attempts: 3
      retry_delay_milliseconds: 1000
      retry_max_jitter_milliseconds: 100
      retry_delay_type: "back-off"
    source:
    ......  

Rate Limiter Middleware

KubeMQ targets support a Rate Limiting of target executions.

Rate Limiter middleware settings values:

Property	Description	Possible Values
rate_per_second	how many executions per second will be allowed	0 - no limitation
		1 - n integer times per second

An example for 100 executions per second:

bindings:
  - name: sample-binding 
    properties: 
      rate_per_second: 100
    source:
    ......  

Source

Source section contains source configuration for Binding as follows:

Property	Description	Possible Values
name	sources name (will show up in logs)	string without white spaces
kind	source kind type	kubemq.queue
		kubemq.query
		kubemq.query-stream
		kubemq.command
		kubemq.events
		kubemq.events-store
properties	an array of key/value setting for source connection	see above

Target

Target section contains the target configuration for Binding as follows:

Property	Description	Possible Values
name	targets name (will show up in logs)	string without white spaces
kind	source kind type	type-of-target
properties	an array of key/value set for target connection	see above