webconv

README

XML & JSON 2-Way Converter

This work is done just in a test case. So, it have some pitfails and flaws.

Installing

Just execute in terminal git clone github.com/awsom82/happyconv && cd happyconv and run command go build ./cmd/webconv && ./webconv.

This will run conversion http service at port 8080.

Using

After your run this app, you should able to send any JSON or XML file to http://localhost:8080/.

Notice, there no specific path for JSON or XML. The application will detect an input type of file by a mime-type header, or if it lacks that info. It will try to detect that by file signature MIME Sniffing

Configuration

You can just type ./webconv --help to get help message.

Usage of ./webconv:

  -hostname string
  	Bind server address (default "localhost")
  -keep-alive
  	HTTP Keep-Alive
  -port int
  	Port number (default 8080)
  -rate float
  	Rate limiter (default 200000)
  -read-timeout duration
  	HTTP Read timeout (default 5s)
  -ttl duration
  	Rate limiter TTL (default 5s)
  -write-timeout duration
  	HTTP Write timeout (default 10s)

Examples

$ http :8080 Content-type:application/xml < example.xml
$ http :8080 Content-type:application/json < example.json

Testing & Benchmarking

Test with go test -v .

and run benchmarks go test -bench . -benchmem -parallel 24 -cpu 8

BenchmarkParallelTestServeHTTP-8   	   43688	     45218 ns/op	   20037 B/op	     154 allocs/op
BenchmarkJson2Xml-8                	   41541	     33855 ns/op	   41864 B/op	     211 allocs/op
BenchmarkXml2Json-8                	   31566	     38070 ns/op	   19704 B/op	     406 allocs/op
PASS
ok  	github.com/awsom82/happyconv	5.550s

wrk

Good start point for testing is run app with next keys:

./webconv -rate 3e5 -read-timeout 0.5s -write-timeout 1.3s -keep-alive 1.

Use wrk -t12 -c400 -d30s -s./wrk-xml.lua http://localhost:8080/ for simply load test

Beat next results

JSON 104.00k rps is best and XML 14.05k rps is best result

json-small

wrk -t1 -c9000 -d30s -s./wrk-json.lua http://localhost:8080/
Running 30s test @ http://localhost:8080/
  1 threads and 9000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    33.93ms   70.66ms   1.07s    95.39%
    Req/Sec    37.26k    26.31k  104.00k    64.34%
  925155 requests in 30.06s, 292.92MB read
  Socket errors: connect 7980, read 5, write 0, timeout 1
Requests/sec:  30774.50
Transfer/sec:      9.74MB

books.xml

wrk -t1 -c9400 -d30s -s./wrk-xml.lua http://localhost:8080/
Running 30s test @ http://localhost:8080/
  1 threads and 9400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    80.75ms   95.77ms   1.93s    85.83%
    Req/Sec    12.70k     0.87k   14.05k    97.98%
  376629 requests in 30.07s, 1.24GB read
  Socket errors: connect 8380, read 0, write 0, timeout 4
  Non-2xx or 3xx responses: 379
Requests/sec:  12524.26
Transfer/sec:     42.39MB

Documentation

Index

• Constants
• Variables
• func NewServer(conf *Config) *http.Server
• func WebconvHandler(w http.ResponseWriter, r *http.Request)
• func WebconvLogMiddleware(next http.Handler) http.Handler
• type Config
  • func NewConfig() *Config
• type Converter
  • func NewConv() *Converter
  • func (c *Converter) CopyInput(r *http.Request)
  • func (c *Converter) MakeReply(w http.ResponseWriter, err error)
  • func (c *Converter) SwapFormat() error
• type MimeType
  • func (m *MimeType) DetectContentType(ct string, body []byte)
  • func (m *MimeType) FromString(ct string)
  • func (m *MimeType) String() string

Constants

const (
	InfoColor    = "\033[1;34m%s\033[0m" + endString
	WarningColor = "\033[1;33m%s\033[0m" + endString
)

Log colors

Variables

var (
	// ErrConversion is a converter error
	ErrConversion = errors.New("webconv: conversion error, possible mailformed payload; check logs")
	// ErrUnsupported is error type to flag unsupported formats
	ErrUnsupported = errors.New("webconv: unsupported format")
)

Functions

func NewServer

func NewServer(conf *Config) *http.Server

NewServer creates new server and limiter

func WebconvHandler

func WebconvHandler(w http.ResponseWriter, r *http.Request)

WebconvHandler a http.handler function

func WebconvLogMiddleware

func WebconvLogMiddleware(next http.Handler) http.Handler

WebconvLogMiddleware logs http requests

Types

type Config

type Config struct {
	Hostname     string
	Port         int // pflag doesnt support uint16 out of the box
	RateLimit    float64
	RateLimitTTL time.Duration
	KeepAlive    bool
	ReadTimeout  time.Duration
	WriteTimeout time.Duration
}

func NewConfig

func NewConfig() *Config

type Converter

type Converter struct {
	Payload           bytes.Buffer
	ContentType       MimeType
	Result            bytes.Buffer
	ResultContentType MimeType
}

Converter holds payload and conversion results

func NewConv

func NewConv() *Converter

NewConv returns new Converter struct

func (*Converter) CopyInput

func (c *Converter) CopyInput(r *http.Request)

CopyInput creates copy of received data and sets content-type

func (*Converter) MakeReply

func (c *Converter) MakeReply(w http.ResponseWriter, err error)

MakeReply returns forms output reply from webserver

func (*Converter) SwapFormat

func (c *Converter) SwapFormat() error

SwapFormat fill Input.Result if input data is Unsupported they return raw data back (TODO: maybe is better to return 415 Unsupported Media Type?)

type MimeType

type MimeType uint8

MimeType represents Content-Type

const (
	// Unsupported used for unsupported content-types
	Unsupported MimeType = iota
	// JSON represents Content-Type: application/json
	JSON
	// XML represents Content-Type: application/xml and it's variations
	XML
)

func (*MimeType) DetectContentType

func (m *MimeType) DetectContentType(ct string, body []byte)

DetectContentType detects MIME-Type by reading it from header, if this sections is missing. Then we try to detect it by MIME Sniff algorithm (https://mimesniff.spec.whatwg.org/) this is embedded algorithm in net/http std package

func (*MimeType) FromString

func (m *MimeType) FromString(ct string)

FromString sets MimeType from string It receives a content-type, removes charset part if they presence in ct Ans sets proper value

func (*MimeType) String

func (m *MimeType) String() string

String return mimetype string