README
¶
simdjson
JSON parsing for Go that finds the whole document's structure in a few vector passes, then walks that instead of the bytes. Built on simd.go.
No cgo, and it runs the same on amd64, arm64, riscv64, s390x, ppc64le and loong64. The established Go port, minio/simdjson-go, is amd64 with AVX2 and hand-written assembly — and on amd64 it is faster than this, by 1.3–1.8×. The trade this package makes is portability for speed, and the numbers say so plainly.
go get github.com/sebishogun/simdjson
doc, err := simdjson.Parse(data)
if err != nil {
return err
}
name := doc.Get("user", "name").String()
age := doc.Get("user", "age").Int()
doc.Get("items").ForEach(func(v simdjson.Value) bool {
total += v.Key("score").Float()
return true
})
Numbers
Zen 5, worse of two runs of six, against encoding/json unmarshalling into
map[string]any and reading the same field:
| encoding/json | Scan |
|
|---|---|---|
| get 1 field, 100 items | 0.116 ms | 7.97× |
| get 1 field, 1,000 items | 0.999 ms | 6.77× |
| get 1 field, 10,000 items | 11.77 ms | 7.83× |
Against minio/simdjson-go
minio/simdjson-go is the established Go port of this idea, in hand-written AVX2 assembly. Same machine, same documents, worse of two runs of six at a load average under 2:
| get one field | encoding/json | minio | this | vs stdlib | vs minio |
|---|---|---|---|---|---|
| 100 items | 0.116 ms | 0.026 ms | 0.015 ms | 7.97× | 1.78× |
| 1,000 items | 0.999 ms | 0.225 ms | 0.148 ms | 6.77× | 1.52× |
| 10,000 items | 11.77 ms | 2.090 ms | 1.504 ms | 7.83× | 1.39× |
Parse alone, 2,000 items: minio 0.425 ms, Scan 0.298 ms — 1.43×.
It was the other way round until v0.2.0. The first release made one vector pass per structural character — six of them — plus quotes and backslashes, then merged eight ascending lists. minio computes the same thing in one fused pass over 64-byte chunks, using a carry-less multiply to turn quote parity into a prefix-XOR, and it was 1.3–1.8× faster.
Measuring that is what produced the fix. simd.IndexAll takes a single byte, so
six delimiters meant six reads of the document; simd.go v1.3.0 added
IndexAllAny, which takes a set and is 2.6–2.9× six separate calls. Eight
passes and a merge became one pass and a filter.
minio is still amd64-only. It requires AVX2 and refuses to run without it. This runs on amd64, arm64, riscv64, s390x, ppc64le and loong64 from one source, because the kernels are generated per target rather than written by hand — and it is now faster on the architecture minio was written for.
Two things in that table are worth reading carefully.
Scan is 6.8–8.0× the standard library across every size, which is the case
this package is for: values out of a document without decoding the rest.
Parse is not faster at all. Validating every value costs about what
encoding/json costs, and this does it and builds an index. If you need the
validation you are better off with the standard library; if you produced the
bytes, Scan is the point.
Walking everything is 0.45×. The standard library decodes in one fused pass; this indexes and then navigates. Reaching into a document is what an index buys, and reading all of it is what it does not.
How it works
Two stages, which is the design simdjson introduced.
Stage one finds every structural character — { } [ ] : , — in one vector
pass each, and works out which quotes really open and close strings rather than
being escaped. A conventional parser reads a byte and branches on what it is,
which is a dependent and unpredictable branch per byte. This makes a handful of
branch-free passes instead.
Stage two walks those positions. A megabyte document might have fifty thousand structural characters, so the second stage sees fifty thousand items rather than a million bytes.
The difficulty is entirely in stage one. A { inside a string is text, and a
" preceded by an odd number of backslashes does not close anything — in
"a\\" the quote follows two backslashes and does close the string, while in
"a\" it follows one and does not. Both are resolved before any position is
interpreted.
Parse or Scan
Parse validates. It checks every value against JSON's grammar and rejects
exactly what encoding/json rejects. Use it for anything from outside.
Scan does not. It builds the index and identifies the root, and skips the
recursive descent that proves the parts you never look at are well-formed.
Malformed input then gives wrong answers rather than errors — nothing reads out
of bounds and nothing panics, but the result is not to be trusted. Use it when
you produced the bytes.
Validation is most of the cost, and skipping work you did not ask for is the whole reason a structural index exists.
Parser reuses its index between documents, which is what a server handling
a stream of payloads wants. Reuse cuts allocation by about 625× — 999 KB to 1.6
KB per parse — and about 4% of the time, because Go's allocator was already
handing back warm memory. The allocation is worth removing; do not expect the
clock to move much.
Correctness
Defined as agreeing with encoding/json, and tested that way: hand-written
cases, 2000 randomised documents built from atoms chosen to collide (structure
inside strings, escaped quotes, escaped backslashes, surrogate pairs), and
fuzzing — 49 million executions, clean.
The fuzzer found four real bugs in its first three minutes, none of which the hand-written tests caught:
| input | bug |
|---|---|
{"":"\x82"} |
invalid UTF-8 returned raw; encoding/json coerces it to U+FFFD |
{"":{"":[{"\x00":0}]}} |
raw control character in a string, which JSON forbids |
{"":"\0"} |
invalid escape accepted — unquote returned a false flag the parse path ignored |
{"":10.} |
strconv.ParseFloat accepts 10.; JSON's grammar does not |
It then found a fifth failure that was in the test: 1E700 is valid JSON
that does not fit a float64. Comparing against Unmarshal, which converts,
made a conversion limit look like a syntax rule. The oracle for accept-or-reject
is json.Valid.
go test ./...
go test -run '^$' -fuzz FuzzAgainstStdlib -fuzztime 60s
What this is not
Not a replacement for encoding/json. No struct unmarshalling, no tags, no
interfaces, no streaming, no encoding. If you want a Go value, use the standard
library.
Not faster at everything. Parse, which validates every value, costs about
what encoding/json costs and builds an index on top; use it for untrusted
input and Scan when you produced the bytes. Walking a whole document is slower
than the standard library's single fused decode. Reaching into a document is
what an index buys; reading all of it is what it does not.
Not zero-copy for strings with escapes. A string with no backslash is returned without copying out of the document; one with an escape is decoded into a new string.
Status
Early, and measured on amd64 only. The simd package underneath is verified on
amd64 and arm64 NEON and under emulation elsewhere.
License
Documentation
¶
Overview ¶
Package simdjson parses JSON by finding the whole document's structure in a few vector passes, then walking that instead of the bytes.
It is built on [simd.go](https://github.com/sebishogun/simd), so it needs no cgo and runs the same way on amd64, arm64, riscv64, s390x, ppc64le and loong64 — unlike the existing Go ports of simdjson, which are amd64 with hand-written assembly.
doc, err := simdjson.Parse(data)
name := doc.Get("user", "name").String()
age := doc.Get("user", "age").Int()
How it works ¶
Two stages, which is the design simdjson introduced.
Stage one finds every structural character — the braces, brackets, colons and commas — in one vector pass each, and works out which quotes really open and close strings rather than being escaped. A conventional parser reads a byte and branches on what it is, which is a dependent and unpredictable branch per byte; this makes eight branch-free passes over the document instead, and eight passes with no branches beat one pass with a branch per byte.
Stage two walks those positions. A document of a megabyte might have fifty thousand structural characters, so the second stage sees fifty thousand items rather than a million bytes.
What it is for ¶
Pulling a few values out of a document, which is most of what JSON is used for and the case encoding/json is worst at — it decodes everything to reach anything. Doc.Get navigates the index without decoding what it passes.
It is not a replacement for encoding/json. There is no struct unmarshalling, no tags, no interfaces, no streaming. If you want a Go value, use the standard library; if you want three fields out of a large payload, this is several times faster.
Example ¶
The case this package is for: a few values out of a document, without decoding the rest of it.
package main
import (
"fmt"
"github.com/sebishogun/simdjson"
)
func main() {
data := []byte(`{
"user": {"name": "ada", "age": 36, "tags": ["math", "engines"]},
"meta": {"page": 1}
}`)
doc, err := simdjson.Parse(data)
if err != nil {
fmt.Println("bad json:", err)
return
}
fmt.Println(doc.Get("user", "name").String())
fmt.Println(doc.Get("user", "age").Int())
fmt.Println(doc.Get("user", "tags").Index(1).String())
}
Output: ada 36 engines
Index ¶
- type Doc
- type Kind
- type Parser
- type Value
- func (v Value) Bool() bool
- func (v Value) Exists() bool
- func (v Value) Float() float64
- func (v Value) ForEach(fn func(Value) bool)
- func (v Value) ForEachKey(fn func(string, Value) bool)
- func (v Value) Index(n int) Value
- func (v Value) Int() int64
- func (v Value) IsNull() bool
- func (v Value) Key(name string) Value
- func (v Value) Kind() Kind
- func (v Value) Len() int
- func (v Value) Raw() []byte
- func (v Value) String() string
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Doc ¶
type Doc struct {
// contains filtered or unexported fields
}
Doc is a parsed document. It holds the input and its structural index; no values are decoded until they are asked for.
func Parse ¶
Parse indexes data and validates its structure.
The returned Doc keeps data — it is not copied, and every string a Value yields points into it unless the string contains an escape.
Example (StringsHideStructure) ¶
Structure inside a string is text, which is the whole difficulty of stage one and is handled before any of it is interpreted.
package main
import (
"fmt"
"github.com/sebishogun/simdjson"
)
func main() {
doc, err := simdjson.Parse([]byte(`{"a":"},{\"b\":2},[","c":1}`))
if err != nil {
fmt.Println("err:", err)
return
}
fmt.Printf("%q\n", doc.Get("a").String())
fmt.Println(doc.Get("c").Int())
}
Output: "},{\"b\":2},[" 1
func Scan ¶
Scan indexes data without validating it.
Parse walks the whole document and checks every value against JSON's grammar, which is what makes it safe for input you did not produce — and it is most of the cost. If the goal is three fields out of a payload your own service just serialised, validating the other nine thousand is work nobody asked for.
Scan skips it. The structural index is still built, so navigation works exactly as it does after Parse; what is gone is the recursive descent that proves the parts you never look at are well-formed.
What that costs ¶
Malformed input gives wrong answers rather than errors. A missing colon, a trailing comma, a number like 10., an invalid escape — all are accepted, and the values around them may come back wrong or absent instead of failing. The index itself is still consistent, so nothing reads out of bounds and nothing panics; the result is simply not to be trusted.
Two things are still checked, because the index cannot be built without them: every string is terminated, and quotes balance. A document that fails either is rejected here too.
Use Parse for anything from outside. Use Scan when you produced the bytes.
func (*Doc) Get ¶
Get walks a path of object keys and returns the value at the end.
A missing key, or a path that runs into a non-object, yields an Invalid Value rather than an error — chaining is the common case and an error at every step would be unusable. Check Value.Exists.
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser parses documents, reusing its index buffers between them.
A server handling many payloads should keep one per goroutine: Parse allocates a fresh index each time, and for a document of a few hundred kilobytes that index is several times the size of the document itself. A Parser reuses it, so the second and later documents allocate almost nothing.
A Parser is not safe for concurrent use.
Example ¶
A Parser reuses its index between documents, which is what a server handling a stream of payloads wants. The Doc it returns is only valid until the next Parse on the same Parser.
package main
import (
"fmt"
"github.com/sebishogun/simdjson"
)
func main() {
var p simdjson.Parser
for _, payload := range [][]byte{
[]byte(`{"id":1}`),
[]byte(`{"id":2}`),
} {
doc, err := p.Parse(payload)
if err != nil {
return
}
fmt.Println(doc.Get("id").Int())
}
}
Output: 1 2
type Value ¶
type Value struct {
// contains filtered or unexported fields
}
Value is one JSON value inside a document.
func (Value) Exists ¶
Exists reports whether the value was found.
Example ¶
A missing key yields a Value that does not exist rather than an error, so a path can be walked without checking every step.
package main
import (
"fmt"
"github.com/sebishogun/simdjson"
)
func main() {
doc, _ := simdjson.Parse([]byte(`{"a":{"b":1}}`))
fmt.Println(doc.Get("a", "b").Exists())
fmt.Println(doc.Get("a", "zzz").Exists())
fmt.Println(doc.Get("nope", "deeper").Exists())
}
Output: true false false
func (Value) ForEach ¶
ForEach calls fn for each element of an array until it returns false.
Example ¶
Iterating an array without building one.
package main
import (
"fmt"
"github.com/sebishogun/simdjson"
)
func main() {
doc, _ := simdjson.Parse([]byte(`{"scores":[10,20,30]}`))
total := int64(0)
doc.Get("scores").ForEach(func(v simdjson.Value) bool {
total += v.Int()
return true
})
fmt.Println(total)
}
Output: 60
func (Value) ForEachKey ¶
ForEachKey calls fn for each field of an object until it returns false.
Example ¶
Iterating an object's fields.
package main
import (
"fmt"
"github.com/sebishogun/simdjson"
)
func main() {
doc, _ := simdjson.Parse([]byte(`{"a":1,"b":2}`))
doc.Root().ForEachKey(func(k string, v simdjson.Value) bool {
fmt.Printf("%s=%d\n", k, v.Int())
return true
})
}
Output: a=1 b=2
func (Value) Key ¶
Key returns the value of a field in an object.
The scan walks the object's structural entries rather than its bytes, so passing over a large nested value costs one bracket match instead of a parse. A missing key gives an Invalid Value; see Value.Exists.
Source Files