openexchange-go-data-gov

README

openexchange-go-data-gov

Demonstration of CQRS pattern in Golang

Problem statement

Service to visualize Federal Commerce News feed.

Solution

We are going to subscribe to Commerce.gov News API stream and poll from it periodically in order to collect updates.

All updates will be streamed into a queue (probably Apache Kafka) then consumed independently from the harvesting process. Upon consumption deduplication will be made then all new articles will be parsed out and persisted into a database.

The special web service will host REST API to query articles:

• All with a pagination

• Facet search by categories

• Wildcard search

Technical requirements

• Application should be written in Golang.

• Application to be hosted in a public cloud (preferably in Google Cloud Engine)

• Following tools are permitted:

  • https://github.com/labstack/echo
  • https://github.com/go-resty/resty
  • https://github.com/buger/jsonparser
  • https://github.com/jinzhu/gorm
  • https://github.com/Shopify/sarama

Building instructions

Local

• Install and configure the latest version of Golang specific to your development environment.

• Install Docker Desktop.

• Login to Docker Hub.

• Building steps:

  • git clone https://github.com/andrewkandzuba/openexchange-go-data-gov

  • cd openexchange-go-data-gov

  • ./build.sh

  • ./ci

• If your environment is Window build like this:

  • CGO_ENABLED=1;CC=x86_64-w64-mingw32-gcc ./build.sh

  • CGO_ENABLED=1;CC=x86_64-w64-mingw32-gcc ./ci.sh

Google Cloud Build

TBD

Components

• config - Configuration helper to init application properties from YAML and/or environment variables.
• connector - Data.gov Commerce New REST API client.
• stream - Commerce API News stream.
• ingress - News Articles ingression service.
• db - News Articles database repository.
• web - Query web service.
• channel - Producer / Consumer API.
• kafka - Producer / Consumer API implementation for Apache Kafka.
• main - Main application to demo how it all works together.
• tests - Continues Integration tests.

The flow

The application is designed based on classical CQRS pattern:

• Data harvesting flows [Data.gov Commerce News API] -> [connector] -> [stream] -> [channel] -> [ingress] -> [db]

• Query flow [db] -> [web] -> [client]

Kubernetes deployment

TBD

ToDos

• Replace default test coverage tool with https://github.com/grosser/go-testcov.
• Switch to MySQL and move the environment's initialization into Terraform or Google Cloud Deployment Manager.
• Externalize timeouts configuration.
• Implement retry and back-off for external API.
• Bug in echo 4.1.15 !!! https://github.com/labstack/echo/issues/1492
• Replace in memory channel based communication [connector] -> [ingress] by Apache Kafka integration.