morganfield

README

Morganfield

Introduction

Morganfield is a simple REST/JSON Information Exchange Gateway. It works by mimicking target services, proxying requests to them, and by forcing JSON message structure via means of marshaling & unmarshaling the messages against structure definitions.

The application has minimal amount of moving parts, following the KISS principle. In ideal case the applications never notice that there is an extra component between them.

Installation

go get github.com/mikkolehtisalo/morganfield
cd $GOPATH/src/github.com/mikkolehtisalo/morganfield

Define the allowed JSON messages in objects.go, for example:

type Auth_1_Session_Post_Request struct {
    Username string `json:"username"`
    Password string `json:"password"`
}

Define the allowed services in services.go, for example:

i, err := Create_Service_Definition(
    "GET",                          // Method
    "/rest/auth/1/session",         // Path (regexp string)
    "http",                         // Incoming service's protocol
    "localhost.localdomain",        // Incoming service's host
    "http",                         // Target service's protocol
    "dev.localdomain",              // Target service's host
    ".*",                           // Client limitation by address (most commonly fqdn)
    true,                           // Forwarding cookies?
    nil,                            // Input JSON
    Auth_1_Session_Get_Response{})  // Output JSON
if err != nil {
    syslog.Errf("%v", err)
    panic(err)
}
Services = append(Services, i)

Build morganfield, and install it:

make
sudo make install

Running as service

make install adds systemd service, so you can control the process by systemctl:

systemctl start morganfield
# Enable at system start
systemctl enable morganfield

The systemd service restarts automatically. Adding GOMAXPROCS should not be necessarily, the application attempts to set it automatically to runtime.NumCPU().

SELinux

A simple SELinux policy is included. To install it, use make:

sudo make selinux

Enabling TLS

Morganfield supports TLS for both in and outcoming connections. In order to enable TLS for incoming connections:

• Modify service definitions accordingly for https protocol
• Add server's certificate (/opt/morganfield/etc/morganfield.crt) and private key (/opt/morganfield/etc/morganfield.key)
• Edit morganfield.go. Change http.Server's Addr to :443, and switch the server.ListenAndServe to server.ListenAndServeTLS:

func main() {
    // ...
    server := &http.Server{
            Addr:           ":443", // <= Change the port
            // ...
        }
    //syslog.Errf("%v", server.ListenAndServe())
    syslog.Errf("%v", server.ListenAndServeTLS("/opt/morganfield/etc/morganfield.crt", "/opt/morganfield/etc/morganfield.key"))
}

To enable TLS for outgoing connections:

• Modify service definitions accordingly for https protocol
• Add CA certificate (required to check the identity of target services) /opt/morganfield/etc/extca.crt
• Optionally to offer morganfield's identity, add certificate (/opt/morganfield/etc/morganfield.crt) and private key (/opt/morganfield/etc/morganfield.key)

Effects of filtering

The JSON marshaling works only for well formed JSON. Any parsing failures stop handling requests further.

Construct	Description	Handling
'...'	Single quoted strings	Parsing failure
\xAB	Hex integer literals	Converted to unicode
\012	Octal integer literals	Converted to unicode
0xAB	Hex integer literals	Parsing failure
012	Octal integer literals	Parsing failure
[0,,2]	Elisions	Parsing failure
[1,2,3,]	Trailing commas	Parsing failure
{foo:"bar"}	Unquoted property names	Parsing failure
//comment	Code comment	Parsing failure
(...)	Grouping parentheses	Parsing failure
+.5	Unstrict decimal formats	Parsing failure
{	Missing brackets	Parsing failure
{ }}	Mismatched brackets	Parsing failure
	Only whitespace	Parsing failure
"foo":"baz"	Unknown element	Ignored & removed
	Missing elemement	Default value used

Extra validation

Implementing the Validate() function for any object will allow extra validation step to be run between unmarshaling and marshaling object. For example:

func (a *Auth_1_Session_Post_Request) Validate() {
    // ... perform extra validation
    // ... change data
    // ... or perhaps panic
}

Logging

Logging is done to syslog facility DAEMON. Request log messages are valid JSON.

Forwarding logs to something like ELK is highly recommended. Please note that your syslog implementation has to support large (>1k) packets.

Performance

In short, morganfield can be easily scaled up to thousands of requests per second. Most usually the bottleneck will be the target services.

In quick testing against simple GET service (Jira login information), and POST service (login to Jira) results looked like the following:

• Intel(R) Core(TM) i5-4570T CPU @ 2.90GHz
• Fedora 20 (64-bit)
• Jira 6.3 (default settings, incl. HSQL database) in KVM virtual machine (1 vcpu)
• GOMAXPROCS=4 ./morganfield
• 20 concurrent users
• GET => 250 req/s (avg 26ms)
• POST => 45 req/s (avg 476ms)
• Jira VM maxed out allotted CPU
• pprof on morganfield shows no locking issues, CPU usage is evenly distributed among network io, JSON marshaling / reflection, and memory allocations.

Credits

Created by Mikko Lehtisalo mikko.lehtisalo@gmail.com.

Frequently Asked Questions

Q: Why REST/JSON? Why no insert favorite technology here?

A: Today approximately 2/3 of new applications implement REST/JSON API. Parsing JSON is technically must simpler and less error prone than, say, XML. That is not to say the alternatives do not have their uses.

Q: Why the settings are hard coded?

A: Simplicity. Bolting down the configuration in order to protect them against changes.

Documentation

Index

• Variables
• func Handler(w http.ResponseWriter, r *http.Request)
• func Marshal(in interface{}) (string, error)
• func Setup_services()
• func UnMarshal(in string, out interface{}) (interface{}, error)
• type Auth_1_Session_Get_Response
• type Auth_1_Session_Post_Request
  • func (a *Auth_1_Session_Post_Request) Validate()
• type Auth_1_Session_Post_Response
• type RequestLog
  • func RequestLog_from_request(r *http.Request) RequestLog
  • func (reqlog *RequestLog) Log()
• type Service_Definition
  • func Create_Service_Definition(method string, uri string, intprotocol string, inthost string, ...) (Service_Definition, error)
  • func (s Service_Definition) Matches_Request(r *http.Request) bool
• type Validator

Variables

var (
	Services []Service_Definition = []Service_Definition{}
)

Functions

func Handler

func Handler(w http.ResponseWriter, r *http.Request)

Handler for all requests

func Marshal

func Marshal(in interface{}) (string, error)

Marshals any object into string

func Setup_services

func Setup_services()

func UnMarshal

func UnMarshal(in string, out interface{}) (interface{}, error)

Unmarshals any string into object

Types

type Auth_1_Session_Get_Response

type Auth_1_Session_Get_Response struct {
	Self      string `json:"self"`
	Name      string `json:"name"`
	LoginInfo struct {
		FailedLoginCount    int    `json:"failedLoginCount"`
		LoginCount          int    `json:"loginCount"`
		LastFailedLoginTime string `json:"lastFailedLoginTime"`
		PreviousLoginTime   string `json:"previousLoginTime"`
	} `json:"loginInfo"`
}

type Auth_1_Session_Post_Request

type Auth_1_Session_Post_Request struct {
	Username string `json:"username"`
	Password string `json:"password"`
}

func (*Auth_1_Session_Post_Request) Validate

func (a *Auth_1_Session_Post_Request) Validate()

type Auth_1_Session_Post_Response

type Auth_1_Session_Post_Response struct {
	Session struct {
		Name  string `json:"name"`
		Value string `json:"value"`
	} `json:"session"`
	LoginInfo struct {
		FailedLoginCount    int    `json:"failedLoginCount"`
		LoginCount          int    `json:"loginCount"`
		LastFailedLoginTime string `json:"lastFailedLoginTime"`
		PreviousLoginTime   string `json:"previousLoginTime"`
	} `json:"loginInfo"`
}

type RequestLog

type RequestLog struct {
	Method        string
	Host          string
	Path          string
	Proto         string
	ContentLength int64
	ContentType   string `json:"Content-Type"`
	UserAgent     string `json:"User-Agent"`
	RemoteAddr    string
	InCookies     []string
	OutCookies    []string
	InJson        string
	OutJson       string
	Status        int
}

func RequestLog_from_request

func RequestLog_from_request(r *http.Request) RequestLog

Initialize new RequestLog and fill in some details from request

func (*RequestLog) Log

func (reqlog *RequestLog) Log()

Print object to syslog as JSON string

type Service_Definition

type Service_Definition struct {
	Method            string
	URI               *regexp.Regexp
	Internal_Protocol string
	Internal_Host     string
	External_Protocol string
	External_Host     string
	Caller            *regexp.Regexp // checked against RemoteAddr, form host:port!
	SetCookies        bool
	In_Object         interface{}
	Out_Object        interface{}
}

Method: GET | PUT | POST | DELETE Uri: Regexp for path Internal_Protocol: http | https Internal_Host: Address used to connect to morganfield External_Protocol: http | https External_Host: Address used to connect to remote REST service Caller: Regexp for checking source of requests SetCookies: Should the cookie data be copied back and forth In_Object: Input JSON object (see objects.go) Out_Object: Output JSON object (see objects.go)

func Create_Service_Definition

func Create_Service_Definition(
	method string,
	uri string,
	intprotocol string,
	inthost string,
	extprotocol string,
	exthost string,
	caller string,
	setcookies bool,
	inobj interface{},
	outobj interface{}) (Service_Definition, error)

func (Service_Definition) Matches_Request

func (s Service_Definition) Matches_Request(r *http.Request) bool

Service is considered to match request, when the following match: method, url path, target host, remote address

type Validator

type Validator interface {
	Validate()
}

Optional validation functionality