json

README

pkg/json

An alternative JSON decoder for Go.

Features

pkg/json aims to be a drop in replacement for encoding/json. It features:

• json.Scanner which, when provided an external buffer, does not allocate.
• io.Reader friendly; you don't need to buffer your input in memory.
• json.Decoder.Token() replacement that is 2-3x faster than encoding/json.
• json.Decoder.NextToken() is almost allocation free.

Is it faster than fastjson/ultrajson/megajson/fujson?

Honestly, I don't know. I have some benchmarks that show that pkg/json is faster than encoding/json for tokenisation, but this package isn't finished yet.

The Decoder.Token API is between 2-3x faster than encoding/json.Decoder.Token:

BenchmarkDecoderToken/pkgjson/canada.json.gz-16                       51          21387427 ns/op         105.25 MB/s     4402975 B/op     222279 allocs/op
BenchmarkDecoderToken/encodingjson/canada.json.gz-16                  18          63962471 ns/op          35.19 MB/s    17740379 B/op     889106 allocs/op
BenchmarkDecoderToken/pkgjson/citm_catalog.json.gz-16                235           5110849 ns/op         337.95 MB/s      966008 B/op      81995 allocs/op
BenchmarkDecoderToken/encodingjson/citm_catalog.json.gz-16            61          19635188 ns/op          87.96 MB/s     5665660 B/op     324799 allocs/op
BenchmarkDecoderToken/pkgjson/twitter.json.gz-16                     495           2424694 ns/op         260.45 MB/s      768370 B/op      38992 allocs/op
BenchmarkDecoderToken/encodingjson/twitter.json.gz-16                 98          12028296 ns/op          52.50 MB/s     3660269 B/op     187815 allocs/op
BenchmarkDecoderToken/pkgjson/code.json.gz-16                         75          16111648 ns/op         120.44 MB/s     4304252 B/op     320235 allocs/op
BenchmarkDecoderToken/encodingjson/code.json.gz-16                    15          75464079 ns/op          25.71 MB/s    23355964 B/op    1319125 allocs/op
BenchmarkDecoderToken/pkgjson/example.json.gz-16                   23210             51699 ns/op         251.92 MB/s       16032 B/op        914 allocs/op
BenchmarkDecoderToken/encodingjson/example.json.gz-16               4767            257169 ns/op          50.64 MB/s       82416 B/op       4325 allocs/op
BenchmarkDecoderToken/pkgjson/sample.json.gz-16                     1411            849803 ns/op         809.00 MB/s      213776 B/op       5081 allocs/op
BenchmarkDecoderToken/encodingjson/sample.json.gz-16                 345           3550453 ns/op         193.63 MB/s      759686 B/op      26643 allocs/op

The alternative Decoder.NextToken API is between 8-10x faster than encoding/json.Decoder.Token while producing virtually no allocations:

BenchmarkDecoderNextToken/pkgjson/canada.json.gz-16                  222           5157299 ns/op         436.48 MB/s         152 B/op          3 allocs/op
BenchmarkDecoderNextToken/encodingjson/canada.json.gz-16              18          67566098 ns/op          33.32 MB/s    17740510 B/op     889106 allocs/op
BenchmarkDecoderNextToken/pkgjson/citm_catalog.json.gz-16            510           2260514 ns/op         764.08 MB/s         152 B/op          3 allocs/op
BenchmarkDecoderNextToken/encodingjson/citm_catalog.json.gz-16        61          20026083 ns/op          86.25 MB/s     5665620 B/op     324799 allocs/op
BenchmarkDecoderNextToken/pkgjson/twitter.json.gz-16                1032           1093466 ns/op         577.53 MB/s         168 B/op          4 allocs/op
BenchmarkDecoderNextToken/encodingjson/twitter.json.gz-16             93          12548028 ns/op          50.33 MB/s     3660291 B/op     187815 allocs/op
BenchmarkDecoderNextToken/pkgjson/code.json.gz-16                    211           5322161 ns/op         364.60 MB/s         264 B/op          6 allocs/op
BenchmarkDecoderNextToken/encodingjson/code.json.gz-16                14          78195551 ns/op          24.82 MB/s    23355961 B/op    1319125 allocs/op
BenchmarkDecoderNextToken/pkgjson/example.json.gz-16               49245             22873 ns/op         569.41 MB/s         168 B/op          4 allocs/op
BenchmarkDecoderNextToken/encodingjson/example.json.gz-16           4500            269313 ns/op          48.36 MB/s       82416 B/op       4325 allocs/op
BenchmarkDecoderNextToken/pkgjson/sample.json.gz-16                 2032            553729 ns/op        1241.57 MB/s        1160 B/op          9 allocs/op
BenchmarkDecoderNextToken/encodingjson/sample.json.gz-16             332           3574484 ns/op         192.33 MB/s      759685 B/op      26643 allocs/op

Decoding into an interface{} this package is 40-50% faster than encoding/json:

BenchmarkDecoderDecodeInterfaceAny/pkgjson/canada.json.gz-16                  43          27181309 ns/op          82.82 MB/s     8747225 B/op     281408 allocs/op
BenchmarkDecoderDecodeInterfaceAny/encodingjson/canada.json.gz-16             30          36930290 ns/op          60.95 MB/s    20647699 B/op     392553 allocs/op
BenchmarkDecoderDecodeInterfaceAny/pkgjson/citm_catalog.json.gz-16           156           7635508 ns/op         226.21 MB/s     5197871 B/op      89673 allocs/op
BenchmarkDecoderDecodeInterfaceAny/encodingjson/citm_catalog.json.gz-16       73          15766855 ns/op         109.55 MB/s     9410808 B/op      95496 allocs/op
BenchmarkDecoderDecodeInterfaceAny/pkgjson/twitter.json.gz-16                327           3664618 ns/op         172.33 MB/s     2130822 B/op      30182 allocs/op
BenchmarkDecoderDecodeInterfaceAny/encodingjson/twitter.json.gz-16           175           6800730 ns/op          92.86 MB/s     4359944 B/op      31775 allocs/op
BenchmarkDecoderDecodeInterfaceAny/pkgjson/code.json.gz-16                    64          17302914 ns/op         112.15 MB/s     7331655 B/op     232059 allocs/op
BenchmarkDecoderDecodeInterfaceAny/encodingjson/code.json.gz-16               49          23759564 ns/op          81.67 MB/s    12332748 B/op     271292 allocs/op
BenchmarkDecoderDecodeInterfaceAny/pkgjson/example.json.gz-16              15920             76561 ns/op         170.11 MB/s       50988 B/op        739 allocs/op
BenchmarkDecoderDecodeInterfaceAny/encodingjson/example.json.gz-16          8670            139230 ns/op          93.54 MB/s       82840 B/op        782 allocs/op
BenchmarkDecoderDecodeInterfaceAny/pkgjson/sample.json.gz-16                1084           1119477 ns/op         614.12 MB/s      399983 B/op       5542 allocs/op
BenchmarkDecoderDecodeInterfaceAny/encodingjson/sample.json.gz-16            217           5418480 ns/op         126.88 MB/s     2697781 B/op       8075 allocs/op

Should I use this?

Right now? No. In the future, maybe.

I've found a bug!

Great! Please follow this two step process:

1. Raise an issue
2. Send a PR (unless I fix it first)

Standing on the shoulders of giants

This project is heavily influenced by Steven Schveighoffer's iopipe and Phil Pearl.

Documentation

Overview

Package json decodes JSON.

Index

• Constants
• type Decoder
  • func NewDecoder(r io.Reader) *Decoder
  • func NewDecoderBuffer(r io.Reader, buf []byte) *Decoder
  • func (d *Decoder) Decode(v interface{}) error
  • func (d *Decoder) NextToken() ([]byte, error)
  • func (d *Decoder) Token() (json.Token, error)
• type Scanner
  • func NewScanner(r io.Reader) *Scanner
  • func (s *Scanner) Error() error
  • func (s *Scanner) Next() []byte

Examples

• Decoder.Decode
• Decoder.NextToken
• Decoder.Token
• Scanner.Next

Constants

const (
	ObjectStart = '{' // {
	ObjectEnd   = '}' // }
	String      = '"' // "
	Colon       = ':' // :
	Comma       = ',' // ,
	ArrayStart  = '[' // [
	ArrayEnd    = ']' // ]
	True        = 't' // t
	False       = 'f' // f
	Null        = 'n' // n
)

Types

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

A Decoder decodes JSON values from an input stream.

func NewDecoder

func NewDecoder(r io.Reader) *Decoder

NewDecoder returns a new Decoder for the supplied Reader r.

func NewDecoderBuffer

func NewDecoderBuffer(r io.Reader, buf []byte) *Decoder

NewDecoderBuffer returns a new Decoder for the supplier Reader r, using the []byte buf provided for working storage.

func (*Decoder) Decode

func (d *Decoder) Decode(v interface{}) error

Decode reads the next JSON-encoded value from its input and stores it in the value pointed to by v.

Example

package main

import (
	"fmt"
	"log"
	"strings"

	"github.com/pkg/json"
)

func main() {
	input := `{"a": 1,"b": 123.456, "c": [null]}`
	dec := json.NewDecoder(strings.NewReader(input))
	var i interface{}
	err := dec.Decode(&i)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%v\n", i)

}

Output:

map[a:1 b:123.456 c:[<nil>]]

func (*Decoder) NextToken

func (d *Decoder) NextToken() ([]byte, error)

NextToken returns a []byte referencing the next logical token in the stream. The []byte is valid until Token is called again. At the end of the input stream, Token returns nil, io.EOF.

Token guarantees that the delimiters [ ] { } it returns are properly nested and matched: if Token encounters an unexpected delimiter in the input, it will return an error.

A valid token begins with one of the following:

{ Object start
[ Array start
} Object end
] Array End
t JSON true
f JSON false
n JSON null
" A string, possibly containing backslash escaped entites.
-, 0-9 A number

Commas and colons are elided.

Example

package main

import (
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/pkg/json"
)

func main() {
	input := `{"a": 1,"b": 123.456, "c": [null]}`
	dec := json.NewDecoder(strings.NewReader(input))
	for {
		tok, err := dec.NextToken()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("%s\n", tok)
	}

}

Output:

{
"a"
1
"b"
123.456
"c"
[
null
]
}

func (*Decoder) Token

func (d *Decoder) Token() (json.Token, error)

Token returns the next JSON token in the input stream. At the end of the input stream, Token returns nil, io.EOF.

Token guarantees that the delimiters [ ] { } it returns are properly nested and matched: if Token encounters an unexpected delimiter in the input, it will return an error.

The input stream consists of basic JSON values—bool, string, number, and null—along with delimiters [ ] { } of type json.Delim to mark the start and end of arrays and objects. Commas and colons are elided.

Note: this API is provided for compatibility with the encoding/json package and carries a significant allocation cost. See NextToken for a more efficient API.

Example

package main

import (
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/pkg/json"
)

func main() {
	input := `{"a": 1,"b": 123.456, "c": [null]}`
	dec := json.NewDecoder(strings.NewReader(input))
	for {
		tok, err := dec.Token()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("%v\n", tok)
	}

}

Output:

{
a
1
b
123.456
c
[
<nil>
]
}

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner implements a JSON scanner as defined in RFC 7159.

func NewScanner

func NewScanner(r io.Reader) *Scanner

NewScanner returns a new Scanner for the io.Reader r. A Scanner reads from the supplied io.Reader and produces via Next a stream of tokens, expressed as []byte slices.

func (*Scanner) Error

func (s *Scanner) Error() error

Error returns the first error encountered. When underlying reader is exhausted, Error returns io.EOF.

func (*Scanner) Next

func (s *Scanner) Next() []byte

Next returns a []byte referencing the the next lexical token in the stream. The []byte is valid until Next is called again. If the stream is at its end, or an error has occurred, Next returns a zero length []byte slice.

A valid token begins with one of the following:

{ Object start
[ Array start
} Object end
] Array End
, Literal comma
: Literal colon
t JSON true
f JSON false
n JSON null
" A string, possibly containing backslash escaped entites.
-, 0-9 A number

Example

package main

import (
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/pkg/json"
)

func main() {
	input := `{"a": 1,"b": 123.456, "c": [null]}`
	sc := json.NewScanner(strings.NewReader(input))
	for {
		tok := sc.Next()
		if len(tok) < 1 {
			break
		}
		fmt.Printf("%s\n", tok)
	}
	if err := sc.Error(); err != nil && err != io.EOF {
		log.Fatal(err)
	}

}

Output:

{
"a"
:
1
,
"b"
:
123.456
,
"c"
:
[
null
]
}