lile

README

Lile is a application generator (think create-react-app, rails new or django startproject) for gRPC services in Go and a set of tools/libraries.

The primary focus of Lile is to remove the boilerplate when creating new services by creating a basic structure, test examples, Dockerfile, Makefile etc.

Lile comes with basic pre setup with pluggable options for things like...

• Metrics (e.g. Prometheus)
• Tracing (e.g. Zipkin)
• PubSub (e.g. Google PubSub)
• Service Discovery

Installation

Installing Lile is easy, using go get you can install the cmd line app to generate new services and the required libraries. First you'll need Google's Protocol Buffers installed.

$ brew install protobuf
$ go get -u github.com/lileio/lile/...

Getting Started

To generate a new service, run lile new with a short folder path.

$ lile new lileio/users

Guide

• Creating a Service
• Service Definition
• Generating RPC Methods
• Running and Writing Tests
• Using the Generated cmds
• Adding your own cmds

Creating a Service

Lile comes with a 'generator' to quickly generate new Lile services.

Lile follows Go's conventions around $GOPATH (see How to Write Go) and is smart enough to parse your new service's name to create the service in the right place.

If your Github username was lileio and you wanted to create a new service for posting to Slack you might use the following command.

lile new lileio/slack

This will create a project in $GOPATH/src/github.com/lileio/slack

Service Definitions

Lile creates gRPC and therefore uses protocol buffers as the language for describing the service methods, the requests and responses.

I highly recommend reading the Google API Design docs for good advice around general naming of RPC methods and messages and how they might translate to REST/JSON, via the gRPC gateway

An example of a service definition can be found in the Lile example project account_service

service AccountService {
  rpc List (ListAccountsRequest) returns (ListAccountsResponse) {}
  rpc GetById (GetByIdRequest) returns (Account) {}
  rpc GetByEmail (GetByEmailRequest) returns (Account) {}
  rpc AuthenticateByEmail (AuthenticateByEmailRequest) returns (Account) {}
  rpc GeneratePasswordToken (GeneratePasswordTokenRequest) returns (GeneratePasswordTokenResponse) {}
  rpc ResetPassword (ResetPasswordRequest) returns (Account) {}
  rpc ConfirmAccount (ConfirmAccountRequest) returns (Account) {}
  rpc Create (CreateAccountRequest) returns (Account) {}
  rpc Update (UpdateAccountRequest) returns (Account) {}
  rpc Delete (DeleteAccountRequest) returns (google.protobuf.Empty) {}
}

Generating RPC Methods

By default Lile will create a example RPC method and a simple message for request and response.

syntax = "proto3";
option go_package = "github.com/lileio/slack";
package slack;

message Request {
  string id = 1;
}

message Response {
  string id = 1;
}

service Slack {
  rpc Read (Request) returns (Response) {}
}

Let's modify this to be a real service and add our own method.

We're going to create an Announce method that will announce a message to a Slack room.

We're assuming that the Slack team and authentication is already handled by the services configuration, so a user of our service only needs to provide a room and their message. The service is going to send the special Empty response, since we only need to know if an error occurred and don't need to know anything else.

Our proto file now looks like this...

syntax = "proto3";
option go_package = "github.com/lileio/slack";
import "google/protobuf/empty.proto";
package slack;

message AnnounceRequest {
  string channel = 1;
  string msg = 2;
}

service Slack {
  rpc Announce (AnnounceRequest) returns (google.protobuf.Empty) {}
}

We now run the protoc tool with our file and the Lile method generator plugin.

protoc -I . slack.proto --lile-server_out=. --go_out=plugins=grpc:$GOPATH/src

Handily, Lile provides a Makefile with each project that has a proto build step already configured. So we can just run that.

make proto

We can see that Lile will create two files for us in the server directory.

$ make proto
protoc -I . slack.proto --lile-server_out=. --go_out=plugins=grpc:$GOPATH/src
2017/07/12 15:44:01 [Creating] server/announce.go
2017/07/12 15:44:01 [Creating test] server/announce_test.go

Let's take a look at the announce.go file that's created for us.

package server

import (
    "errors"

    "github.com/golang/protobuf/ptypes/empty"
    "github.com/lileio/slack"
    context "golang.org/x/net/context"
)

func (s SlackServer) Announce(ctx context.Context, r *slack.AnnounceRequest) (*empty.Empty, error) {
  return nil, errors.New("not yet implemented")
}

We can now fill in this generated method with the correct implementation. But let's start with a test!

Running & Writing Tests

When you generate an RPC method with Lile a counterpart test file is also created. For example, given our announce.go file, Lile will create announce_test.go in the same directory.

This should look something like the following..

package server

import (
	"testing"

	"github.com/lileio/slack"
	"github.com/stretchr/testify/assert"
	context "golang.org/x/net/context"
)

func TestAnnounce(t *testing.T) {
	ctx := context.Background()
	req := &slack.AnnounceRequest{}

	res, err := cli.Announce(ctx, req)
	assert.Nil(t, err)
	assert.NotNil(t, res)
}

You can now run the tests using the Makefile and running make test...

$ make test
=== RUN   TestAnnounce
--- FAIL: TestAnnounce (0.00s)
        Error Trace:    announce_test.go:16
        Error:          Expected nil, but got: &status.statusError{Code:2, Message:"not yet implemented", Details:[]*any.Any(nil)}
        Error Trace:    announce_test.go:17
        Error:          Expected value not to be nil.
FAIL
coverage: 100.0% of statements
FAIL    github.com/lileio/slack/server  0.011s
make: *** [test] Error 2

Our test failed because we haven't implemented our method, at the moment we're returning an error of "unimplemented" in our method.

Let's implement the Announce method in announce.go, here's an example using nlopes' slack library.

package server

import (
	"os"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	"github.com/golang/protobuf/ptypes/empty"
	"github.com/lileio/slack"
	sl "github.com/nlopes/slack"
	context "golang.org/x/net/context"
)

var api = sl.New(os.Getenv("SLACK_TOKEN"))

func (s SlackServer) Announce(ctx context.Context, r *slack.AnnounceRequest) (*empty.Empty, error) {
	_, _, err := api.PostMessage(r.Channel, r.Msg, sl.PostMessageParameters{})
	if err != nil {
		return nil, status.Errorf(codes.Internal, err.Error())
	}

	return &empty.Empty{}, nil
}

Let's fill out our testing request and then run our tests again...

package server

import (
	"testing"

	"github.com/lileio/slack"
	"github.com/stretchr/testify/assert"
	context "golang.org/x/net/context"
)

func TestAnnounce(t *testing.T) {
	ctx := context.Background()
	req := &slack.AnnounceRequest{
		Channel: "@alex",
		Msg:     "hellooo",
	}

	res, err := cli.Announce(ctx, req)
	assert.Nil(t, err)
	assert.NotNil(t, res)
}

Now if I run the tests with my Slack token as an ENV variable, I should see a passing test!

$ alex@slack: SLACK_TOKEN=zbxkkausdkasugdk make test
go test -v ./... -cover
?       github.com/lileio/slack [no test files]
=== RUN   TestAnnounce
--- PASS: TestAnnounce (0.32s)
PASS
coverage: 75.0% of statements
ok      github.com/lileio/slack/server  0.331s  coverage: 75.0% of statements
?       github.com/lileio/slack/slack   [no test files]
?       github.com/lileio/slack/slack/cmd       [no test files]
?       github.com/lileio/slack/subscribers     [no test files]

Using the Generated cmds

Lile generates a cmd line application based on cobra when you generate your service. You can extend the app with your own cmds or use the built-in cmds to run the service.

Running the cmd line app without any arguments will print the generated help.

For example go run orders/main.go

up

Running up will run both the RPC server and the pubsub subscribers.

go run orders/main.go

Adding your own cmds

To add your own cmd, you can use the built in generator from cobra which powers Lile's cmds

$ cd orders
$ cobra add import

You can now edit the file generated to create your cmd, cobra will automatically add the cmd's name to the help.

Documentation

Overview

Package lile provides helper methods to quickly create RPC based services that have metrics, tracing and pub/sub support

Index

• func AddStreamInterceptor(sint grpc.StreamServerInterceptor)
• func AddUnaryInterceptor(unint grpc.UnaryServerInterceptor)
• func BaseCommand(serviceName, shortDescription string) *cobra.Command
• func ContextClientInterceptor() grpc.UnaryClientInterceptor
• func Name(n string)
• func NewTestServer(s *grpc.Server) (string, func())
• func Run() error
• func ServeGRPC() error
• func Server(r func(s *grpc.Server))
• func Shutdown()
• func TestConn(addr string) *grpc.ClientConn
• func URLForService(name string) string
• type RegisterImplementation
• type Registry
• type ServerConfig
  • func (c *ServerConfig) Address() string
• type Service
  • func GlobalService() *Service
  • func NewService(n string) *Service

Functions

func AddStreamInterceptor

func AddStreamInterceptor(sint grpc.StreamServerInterceptor)

AddStreamInterceptor adds a stream interceptor to the RPC server

func AddUnaryInterceptor

func AddUnaryInterceptor(unint grpc.UnaryServerInterceptor)

AddUnaryInterceptor adds a unary interceptor to the RPC server

func BaseCommand

func BaseCommand(serviceName, shortDescription string) *cobra.Command

BaseCommand provides the basic flags vars for running a service

func ContextClientInterceptor

func ContextClientInterceptor() grpc.UnaryClientInterceptor

ContextClientInterceptor passes around headers for tracing and linkerd

func Name

func Name(n string)

Name sets the name for the service

func NewTestServer

func NewTestServer(s *grpc.Server) (string, func())

NewTestServer is a helper function to create a gRPC server on a non-network socket and it returns the socket location and a func to call which starts the server

func Run

func Run() error

Run is a blocking cmd to run the gRPC and metrics server. You should listen to os signals and call Shutdown() if you want a graceful shutdown or want to handle other goroutines

func ServeGRPC

func ServeGRPC() error

ServeGRPC creates and runs a blocking gRPC server

func Server

func Server(r func(s *grpc.Server))

Server attaches the gRPC implementation to the service

func Shutdown

func Shutdown()

Shutdown gracefully shuts down the gRPC and metrics servers

func TestConn

func TestConn(addr string) *grpc.ClientConn

TestConn is a connection that connects to a socket based connection

func URLForService

func URLForService(name string) string

URLForService returns a service URL via a registry or a simple DNS name if not available via the registry

Types

type RegisterImplementation

type RegisterImplementation func(s *grpc.Server)

RegisterImplementation allows you to register your gRPC server

type Registry

type Registry interface {
	// Register a service
	Register(s *Service) error
	// Deregister a service
	DeRegister(s *Service) error
	// Get a service by name
	Get(name string) (string, error)
}

Registry is the interface to implement for external registry providers

type ServerConfig

type ServerConfig struct {
	Port int
	Host string
}

ServerConfig is a generic server configuration

func (*ServerConfig) Address

func (c *ServerConfig) Address() string

Address Gets a logical addr for a ServerConfig

type Service

type Service struct {
	ID   string
	Name string

	// Interceptors
	UnaryInts  []grpc.UnaryServerInterceptor
	StreamInts []grpc.StreamServerInterceptor

	// The RPC server implementation
	GRPCImplementation RegisterImplementation
	GRPCOptions        []grpc.ServerOption

	// gRPC and Prometheus endpoints
	Config           ServerConfig
	PrometheusConfig ServerConfig

	// Registry allows Lile to work with external registeries like
	// consul, zookeeper or similar
	Registry Registry

	// Private utils, exposed so they can be useful if needed
	ServiceListener  net.Listener
	GRPCServer       *grpc.Server
	PrometheusServer *http.Server
}

Service is a gRPC based server with extra features

func GlobalService

func GlobalService() *Service

GlobalService returns the global service

func NewService

func NewService(n string) *Service

NewService creates a new service with a given name