micro-starter-kit

README

micro-starter-kit

Microservices starter kit for Golang, aims to be developer friendly.

Overview

What you get

• Monorepo - Sharing Code Between Microservices
• gRPC microservices with REST Gateway
• Input Validation with protoc-gen-validate (PGV)
• Config - Pluggable Sources, Mergeable Config, Safe Recovery
• Customizable Logging
• CRUD Example using GORM
• GORM code gen via protoc-gen-gorm
• Dependency injection Container
• One Step build/publish/deploy with ko
• BuildInfo with govvv
• Linting with GolangCI linters aggregator
• CICD Pipelines with GitHub Actions
• Kubernetes Matrix Deployment with Kustomize
• Observability
• Service Mesh with Istio
• GraphQL Gateway with gqlgen, rejoiner

Getting Started

Prerequisite

run following go get ... commands outside this project root and $GOPATH
if you get error, try setting export GO111MODULE=on befor running go get ...

Lets build and install grpc pre-loaded Micro CLI from here instead of default Micro CLI

# fetch micro into $GOPATH
# build and install your own grpc pre-loaded micro-cli
# go get github.com/micro/micro

# go lang  build/publish/deploy tool
go get github.com/google/ko/cmd/ko
# go better build tool
go get github.com/ahmetb/govvv
# for static check/linter
go get github.com/golangci/golangci-lint/cmd/golangci-lint
# for mac, use brew to install protobuf
brew install protobuf
# GUI Client for GRPC Services
brew cask install bloomrpc
# k8s tool similar to helm
brew install kustomize

# fetch protoc plugins into $GOPATH
go get github.com/golang/protobuf/{proto,protoc-gen-go}
go get github.com/micro/protoc-gen-micro
# go get -u github.com/envoyproxy/protoc-gen-validate
# go get -u github.com/infobloxopen/protoc-gen-gorm

Installing PGV can currently only be done from source:

go get -d github.com/envoyproxy/protoc-gen-validate
cd ~/go/src/github.com/envoyproxy/protoc-gen-validate
make build

Installing protoc-gen-gorm can currently only be done from source:

go get -d github.com/infobloxopen/protoc-gen-gorm
cd ~/go/src/github.com/infobloxopen/protoc-gen-gorm
make install

Initial Setup

(optional) setup your workspace from scratch

go mod init github.com/xmlking/micro-starter-kit
mkdir srv api fnc

# scaffold modules
micro new --fqdn="account-srv" --type="srv" --gopath=false \
--alias="account" --plugin=registry=kubernetes srv/account

micro new --fqdn="emailer-srv" --type="srv" --gopath=false \
--alias="emailer"  --plugin=registry=kubernetes:broker=nats srv/emailer

micro new --fqdn="greeter-srv" --type="srv" --gopath=false \
--alias="greeter"  --plugin=registry=kubernetes srv/greeter

micro new --fqdn="account-api" --type="api" --gopath=false \
--alias="account" --plugin=registry=kubernetes api/account

Build

make proto
# silence
make -s proto

make build

Run

Database

By default this project use embedded sqlite3 database. if you want to use postgreSQL,

• start postgres via docker-compose command provided below
• uncommend postgres import statement and comment sqlite in plugin.go
• start micro server with --configFile=config.dev.postgres.yaml flag
  i.e., go run srv/account/main.go srv/account/plugin.go --configFile=config.dev.postgres.yaml

# to start postgres in foreground
docker-compose up postgres
# to stop postgres
docker-compose down
# if needed, remove `postgres_data` volume to recreate database next time, when you start.
docker system prune --volumes

Services

Node: --server_address=localhost:5501x --broker_address=localhost:5502x required only when you are behind VPN a.k.a Work From Home

# dev mode
make run-account
# or
make run-account ARGS="--server_address=localhost:55012 --broker_address=localhost:55022"
# or
go run srv/account/main.go srv/account/plugin.go \
--configDir deploy/bases/account-srv/config \
--server_address=localhost:55011 --broker_address=localhost:55021

make run-emailer
# or
make run-emailer ARGS="--server_address=localhost:55011 --broker_address=localhost:55021"

# integration tests for config module via CMD
make run TARGET=demo TYPE=cmd
go run cmd/demo/main.go --help
go run cmd/demo/main.go --database_host=1.1.1.1 --database_port=7777

export APP_ENV=production
go run cmd/demo/main.go

Test

# Run only Unit tests:
make test-emailer
go test -v -short
go test -v -short ./srv/emailer/service
# Run only Integration Tests: Useful for smoke testing canaries in production.
make inte-emailer
go test -v -run Integration ./srv/emailer/service

UAT Test

using micro CLI

micro list services
micro get service account-srv
micro get service emailer-srv

# how to start proxy
micro proxy --protocol=grpc

Test gRPC Directly

## local build has gRPC by default.
# ./build/micro call  account-srv UserService.Create \
#   '{"username": "sumo", "firstName": "sumo", "lastName": "demo", "email": "sumo@demo.com"}'
micro call  account-srv UserService.Create \
'{"username": "sumo", "firstName": "sumo", "lastName": "demo", "email": "sumo@demo.com"}'
micro call account-srv UserService.Create \
'{"username": "sumo", "firstName": "sumo", "lastName": "demo", "email": "sumo@demo.com"}'
micro call account-srv UserService.List '{}'
micro call account-srv UserService.List '{ "limit": 10, "page": 1}'
micro call account-srv UserService.Get '{"id": "UserIdFromList"}'
micro  call account-srv UserService.Exist '{"username": "sumo", "email": "sumo@demo.com"}'
micro call account-srv UserService.Update \
'{"id": "UserIdFromGet", "firstName": "sumoto222","email": "sumo222@demo.com"}'
micro call account-srv UserService.Delete '{ "id": "UserIdFromGet" }'

Test via Micro Web UI

# Start Web UI for testing
micro web

open http://localhost:8082

create new user from Micro Web UI and see if an email is send

{
  "username": "sumo",
  "firstName": "sumo",
  "lastName": "demo",
  "email": "sumo@demo.com"
}

Test via Micro API Gateway

Start API Gateway

Start API Gateway and run REST Client tests

# start local micro (grpc pre-loaded micro)
make run-micro-cmd ARGS="api --enable_rpc=true"
# (or)
go run cmd/micro/main.go  api --enable_rpc=true

# (or) start global micro
micro  api --enable_rpc=true

GitOps

Deploy

Use ko. If you are new to ko check out the ko-demo

Set a registry and make sure you can push to it:

export PROJECT_ID=ngx-starter-kit
export KO_DOCKER_REPO=gcr.io/${PROJECT_ID}

to publish locally set: export KO_DOCKER_REPO=ko.local

Then apply like this:

ko apply -f deploy/

To deploy in a different namespace:

ko -n nondefault apply -f deploy/

Release

This will publish all of the binary components as container images and create a release.yaml

# publish to  docker repo ar KO_DOCKER_REPO
ko resolve -P -f deploy/ > release.yaml
# publish to local docker repo
ko resolve -P -L -f deploy/ > release.yaml

run local image

docker run -it \
-e MICRO_SERVER_ADDRESS=0.0.0.0:8080 \
-e MICRO_BROKER_ADDRESS=0.0.0.0:10001 \
-e MICRO_REGISTRY=mdns \
-e CONFIG_DIR=/var/run/ko \
-e CONFIG_FILE=config.yaml \
-p 8080:8080 -p 10001:10001 ko.local/github.com/xmlking/micro-starter-kit/srv/account

Docker

Docker Build

# build
TYPE=srv
TARGET=account
VERSION=0.0.5-SNAPSHOT
# DOCKER_REGISTRY=gcr.io
DOCKER_CONTEXT_PATH=xmlking
# docker build --force-rm=true --rm=true --no-cache \
docker build --rm \
--build-arg VERSION=$VERSION \
--build-arg TYPE=${TYPE} \
--build-arg TARGET=${TARGET} \
--build-arg DOCKER_REGISTRY=${DOCKER_REGISTRY} \
--build-arg DOCKER_CONTEXT_PATH=${DOCKER_CONTEXT_PATH} \
--build-arg VCS_REF=$(git rev-parse --short HEAD) \
--build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') \
-t ${DOCKER_REGISTRY:+${DOCKER_REGISTRY}/}${DOCKER_CONTEXT_PATH}/${TARGET}-${TYPE}:${VERSION} .

IMANGE_NAME=${DOCKER_REGISTRY:+${DOCKER_REGISTRY}/}${DOCKER_CONTEXT_PATH}/${TARGET}-${TYPE}:${VERSION}

# push
docker push $IMANGE_NAME

# check
docker inspect  $IMANGE_NAME
# remove temp images after build
docker image prune -f
# Remove dangling images
docker rmi $(docker images -f "dangling=true" -q)
# Remove images tagged with vendor=sumo
docker rmi $(docker images -f "label=org.label-schema.vendor=sumo"  -q)
# Remove all <none> layers
docker rmi $(docker images -a|grep "<none>"|awk '$1=="<none>" {print $3}')

Docker Run

run just for testing image...

docker run -it \
-e MICRO_SERVER_ADDRESS=0.0.0.0:8080 \
-e MICRO_BROKER_ADDRESS=0.0.0.0:10001 \
-e MICRO_REGISTRY=mdns \
-p 8080:8080 -p 10001:10001 $IMANGE_NAME

Docker Compose Run

Run complete app suite with docker-compose

docker-compose up consul
docker-compose up account-srv
docker-compose up emailer-srv
docker-compose up gateway
docker-compose up account-api
docker-compose up gateway-api
curl "http://localhost:8081/account/AccountService/list?limit=10"

Kubernetes Run

run just for testing image in k8s...

# account-srv
kubectl run --rm mytest --image=xmlking/account-srv:latest \
--env="MICRO_REGISTRY=kubernetes" \
--env="MICRO_SELECTOR=static" \
--env="MICRO_SERVER_ADDRESS=0.0.0.0:8080" \
--env="MICRO_BROKER_ADDRESS=0.0.0.0:10001" \
--restart=Never -it

# gateway
kubectl run --rm mygateway --image=microhq/micro:kubernetes \
--env="MICRO_REGISTRY=kubernetes" \
--env="MICRO_SELECTOR=static" \
--restart=Never -it \
--command ./micro api

Make

using Makefile

use -n flag for dry-run, -s or '--silent' flag to suppress echoing

# codegen from proto
make proto
make proto TARGET=account
make proto TARGET=account TYPE=api
make proto-account
make proto-account-api
## generate for protos in shared package
make proto TARGET=shared TYPE=.

# unit tests
make test-account
make test-emailer
make test-account-api
make test-config-shared
make test-demo-cmd

# integration tests
make inte-account
make inte-emailer

# run
make run-account
make run-emailer
make run-account-api
make run-micro-cmd ARGS="--api_address=0.0.0.0:8088 api"
make run-demo-cmd

# Lint
make lint-account-srv

# build
make build VERSION=v0.1.1
make build TARGET=account VERSION=v0.1.1
make build TARGET=account TYPE=srv VERSION=v0.1.1
make build TARGET=emailer TYPE=srv VERSION=v0.1.1
make build TARGET=account TYPE=api VERSION=v0.1.1
make build-account VERSION=v0.1.1
make build-account-api VERSION=v0.1.1

# tag and release
make release VERSION=v0.1.1 GITHUB_TOKEN=123...

# build docker image
make docker-account VERSION=v0.1.1
make docker-account-srv VERSION=v0.1.1
make docker TARGET=account VERSION=v0.1.1
make docker TARGET=account TYPE=srv VERSION=v0.1.1
make docker TARGET=account DOCKER_REGISTRY=us.gcr.io DOCKER_CONTEXT_PATH=<MY_PROJECT_ID>/micro-starter-kit

make docker-emailer-srv
make docker-account-api

# build all docker images
make docker
# build all docker images for google cloud
make docker DOCKER_REGISTRY=us.gcr.io DOCKER_CONTEXT_PATH=<MY_PROJECT_ID>/micro-starter-kit

# publish all microservices images
make docker_push
# remove all previous microservices images and any dangling images
make docker_clean

Reference

1. examples - example usage code for micro
2. microhq - a place for prebuilt microservices
3. explorer - which aggregates micro based open source projects
4. micro-plugins extensible micro plugins