Help us shape the future of Go! Take the 2026 Go Developer Survey

simdjson

package module
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 11, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

README

simdjson-go

This is DATA.BET's fork of minio/simdjson-go. NOTICE says which parts were inherited, which were changed here, and under which licence each stands.

The API is the same as upstream's, so moving over is a change of import path:

go get github.com/data-bet/simdjson-go@latest

import "github.com/data-bet/simdjson-go"

What differs from upstream v0.4.5:

  • Parsing is about 35% faster than upstream with the default WithCopyStrings(true) (geometric mean over the test corpus), and ParseND with a reused ParsedJson allocates almost nothing.
  • The read of a parsed document is approximately 32% faster (geometric mean over BenchmarkIterRead). A walk with AdvanceInto is 30-38% faster, and a walk that also reads each value is 24-31% faster. A walk through Object and NextElementBytes is 14-23% faster, through AdvanceIter 43-51% faster, and through ForEach 27-35% faster.
  • Several inputs that upstream mishandled are fixed: nesting is capped at 1024 levels, as in simdjson, instead of being unbounded; a string scan bound that never fired now does; hex escapes with non-hex digits are rejected; and ParseND reuses the ParsedJson it is given. Some malformed documents that upstream accepted are therefore rejected now. On a corrupt tape NextElementBytes returns an error where it overflowed the stack.
  • After DeleteElems, SetNull or Deserialize, reading continues at the correct element. In upstream, Advance and AdvanceIter skip one word too many after a NOP. Because of this, ForEach, Interface, FindKey and the helpers of Array lose or misread elements.
  • Platforms without the amd64 assembly get a portable implementation instead of an error, and arm64 classifies with NEON - see Requirements.
  • Optional UTF-8 validation with WithValidateUTF8(true).
  • Go 1.25 or newer is required.

Introduction

This is a Golang port of simdjson, a high performance JSON parser developed by Daniel Lemire and Geoff Langdale. It makes extensive use of SIMD instructions to achieve parsing performance of gigabytes of JSON per second.

Parsing a document is about 10x faster than Golang's standard package encoding/json, and allocates three to five orders of magnitude less - see Performance vs encoding/json for the measured figures, the machine they were taken on and the command that reproduces them. The two calls do not do the same amount of work, though: Parse builds a tape and stops, while Unmarshal builds a Go value out of every element. benchmarks/ holds a group where both sides are asked for the same thing, and its README explains the asymmetries; on that footing simdjson-go is about 2.7x faster than encoding/json and roughly level with sonic.

How this compares to the C++ simdjson is not measured here.

Documentation

Features

simdjson-go is a validating parser, meaning that it amongst others validates and checks numerical values, booleans etc. Therefore, these values are available as the appropriate int and float64 representations after parsing.

Additionally simdjson-go has the following features:

  • No 4 GB object limit
  • Support for ndjson (newline delimited json)
  • Pure Go (no need for cgo)
  • Object search/traversal.
  • In-place value replacement.
  • Remove object/array members.
  • Serialize parsed JSONas binary data.
  • Re-serialize parts as JSON.

Requirements

There are two implementations of the parser, and which one you get depends on the platform:

build implementation
amd64, CPU with AVX2 and CLMUL AVX2 assembly, and the AVX-512 variant where the CPU has AVX-512F
amd64, CPU without AVX2 or CLMUL none - SupportedCPU() is false and Parse returns an error
arm64 NEON assembly for the stage 1 classification, portable Go for the rest
any other platform, or -tags=noasm, or gccgo portable Go

The portable implementation is a straight translation of the same algorithm and produces byte for byte identical output - the whole corpus has been checked, on both architectures, tape word for tape word. It is slower, since the step the vector units do

  • turning 64 bytes into the masks the rest of stage 1 works on - becomes a table lookup per byte.

arm64 replaces exactly that step with NEON, in stage1_classify_arm64.s, and shares everything else with the portable path: the bit arithmetic after it compiles about as well from Go as it would from assembly, and the string parser's byte scanning already goes through bytes.IndexByte, which is NEON in the runtime. The string parser has no hand-written arm64 version, so arm64 will not match amd64 on string heavy documents.

The two implementations are held together by stage1_generic_diff_amd64_test.go and parse_string_generic_diff_amd64_test.go, which run the portable code against the assembly over the corpus on amd64, and by stage1_classify_arm64_test.go, which does the same the other way round on arm64.

A CPU with both AVX2 and CLMUL is required for the fast path (Haswell from 2013 onwards should do for Intel, for AMD a Ryzen/EPYC CPU (Q1 2017) should be sufficient). This can be checked using the provided SupportedCPU() function.

An amd64 CPU older than that is the one case with no fallback in the default build. Building with -tags=noasm gives it the portable implementation. It is not selected automatically because the check would have to be made per string in the hottest part of stage 2, which measured at over 2% on string heavy documents for every other user.

Serialized data can be deserialized anywhere, on any CPU.

Usage

Run the following command in order to install simdjson-go

go get -u github.com/data-bet/simdjson-go

In order to parse a JSON byte stream, you either call simdjson.Parse() or simdjson.ParseND() for newline delimited JSON files. Both of these functions return a ParsedJson struct that can be used to navigate the JSON object by calling Iter().

The easiest use is to call ForEach() function of the returned ParsedJson.

func main() {
	// Parse JSON:
	pj, err := Parse([]byte(`{"Image":{"URL":"http://example.com/example.gif"}}`), nil)
	if err != nil {
		log.Fatal(err)
	}

	// Iterate each top level element.
	_ = pj.ForEach(func(i Iter) error {
		fmt.Println("Got iterator for type:", i.Type())
		element, err := i.FindElement(nil, "Image", "URL")
		if err == nil {
			value, _ := element.Iter.StringCvt()
			fmt.Println("Found element:", element.Name, "Type:", element.Type, "Value:", value)
		}
		return nil
	})

	// Output:
	// Got iterator for type: object
	// Found element: URL Type: string Value: http://example.com/example.gif
}
Parsing with iterators

Using the type Iter you can call Advance() to iterate over the tape, like so:

for {
    typ := iter.Advance()

    switch typ {
    case simdjson.TypeRoot:
        if typ, tmp, err = iter.Root(tmp); err != nil {
            return
        }

        if typ == simdjson.TypeObject {
            if obj, err = tmp.Object(obj); err != nil {
                return
            }

            e := obj.FindKey(key, &elem)
            if e != nil && elem.Type == simdjson.TypeString {
                v, _ := elem.Iter.StringBytes()
                fmt.Println(string(v))
            }
        }

    default:
        return
    }
}

When you advance the Iter you get the next type currently queued.

Each type then has helpers to access the data. When you get a type you can use these to access the data:

Type Action on Iter
TypeNone Nothing follows. Iter done
TypeNull Null value
TypeString String()/StringBytes()
TypeInt Int()/Float()
TypeUint Uint()/Float()
TypeFloat Float()
TypeBool Bool()
TypeObject Object()
TypeArray Array()
TypeRoot Root()

You can also get the next value as an interface{} using the Interface() method.

Note that arrays and objects that are null are always returned as TypeNull.

The complex types returns helpers that will help parse each of the underlying structures.

It is up to you to keep track of the nesting level you are operating at.

For any Iter it is possible to marshal the recursive content of the Iter using MarshalJSON() or MarshalJSONBuffer(...).

Currently, it is not possible to unmarshal into structs.

Search by path

It is possible to search by path to find elements by traversing objects.

For example:

	// Find element in path.
	elem, err := i.FindElement(nil, "Image", "URL")

Will locate the field inside a json object with the following structure:

{
    "Image": {
        "URL": "value"
    }
}

The values can be any type. The Element will contain the element information and an Iter to access the content.

Parsing Objects

If you are only interested in one key in an object you can use FindKey to quickly select it.

It is possible to use the ForEach(fn func(key []byte, i Iter), onlyKeys map[string]struct{}) which makes it possible to get a callback for each element in the object.

An object can be traversed manually by using NextElement(dst *Iter) (name string, t Type, err error). The key of the element will be returned as a string and the type of the value will be returned and the provided Iter will contain an iterator which will allow access to the content.

There is a NextElementBytes which provides the same, but without the need to allocate a string.

All elements of the object can be retrieved using a pretty lightweight Parse which provides a map of all keys and all elements an a slide.

All elements of the object can be returned as map[string]interface{} using the Map method on the object. This will naturally perform allocations for all elements.

Parsing Arrays

Arrays in JSON can have mixed types.

It is possible to call ForEach(fn func(i Iter)) to get each element.

To iterate over the array with mixed types use the Iter method to get an iterator.

There are methods that allow you to retrieve all elements as a single type, []int64, []uint64, []float64 and []string with AsInteger(), AsUint64(), AsFloat() and AsString().

Number parsing

Numbers in JSON are untyped and are returned by the following rules in order:

  • If there is any float point notation, like exponents, or a dot notation, it is always returned as float.
  • If number is a pure integer and it fits within an int64 it is returned as such.
  • If number is a pure positive integer and fits within a uint64 it is returned as such.
  • If the number is valid number it is returned as float64.

If the number was converted from integer notation to a float due to not fitting inside int64/uint64 the FloatOverflowedInteger flag is set, which can be retrieved using (Iter).FloatFlags() method.

JSON numbers follow JavaScript’s double-precision floating-point format.

  • Represented in base 10 with no superfluous leading zeros (e.g. 67, 1, 100).
  • Include digits between 0 and 9.
  • Can be a negative number (e.g. -10).
  • Can be a fraction (e.g. .5).
  • Can also have an exponent of 10, prefixed by e or E with a plus or minus sign to indicate positive or negative exponentiation.
  • Octal and hexadecimal formats are not supported.
  • Can not have a value of NaN (Not A Number) or Infinity.

Parsing NDJSON stream

Newline delimited json is sent as packets with each line being a root element.

Here is an example that counts the number of "Make": "HOND" in NDJSON similar to this:

{"Age":20, "Make": "HOND"}
{"Age":22, "Make": "TLSA"}

func findHondas(r io.Reader) {
	var nFound int

	// Communication
	reuse := make(chan *simdjson.ParsedJson, 10)
	res := make(chan simdjson.Stream, 10)

	simdjson.ParseNDStream(r, res, reuse)
	// Read results in blocks...
	for got := range res {
		if got.Error != nil {
			if got.Error == io.EOF {
				break
			}
			log.Fatal(got.Error)
		}

		var result int
		var elem *Element
		err := got.Value.ForEach(func(i Iter) error {
			var err error
			elem, err = i.FindElement(elem, "Make")
			if err != nil {
				return nil
			}
			bts, _ := elem.Iter.StringBytes()
			if string(bts) == "HOND" {
				result++
			}
			return nil
		})
		reuse <- got.Value
	}
	fmt.Println("Found", nFound, "Hondas")
}

More examples can be found in the examples subdirectory and further documentation can be found at godoc.

In-place Value Replacement

It is possible to replace a few, basic internal values. This means that when re-parsing or re-serializing the parsed JSON these values will be output.

Boolean (true/false) and null values can be freely exchanged.

Numeric values (float, int, uint) can be exchanged freely.

Strings can also be exchanged with different values.

Strings and numbers can be exchanged. However, note that there is no checks for numbers inserted as object keys, so if used for this invalid JSON is possible.

There is no way to modify objects, arrays, other than value types above inside each. It is not possible to remove or add elements.

To replace a value, of value referenced by an Iter simply call SetNull, SetBool, SetFloat, SetInt, SetUInt, SetString or SetStringBytes.

Object & Array Element Deletion

It is possible to delete one or more elements in an object.

(*Object).DeleteElems(fn, onlyKeys) will call back fn for each key+ value.

If true is returned, the key+value is deleted. A key filter can be provided for optional filtering. If the callback function is nil all elements matching the filter will be deleted. If both are nil all elements are deleted.

Example:

	// The object we are modifying
	var obj *simdjson.Object

	// Delete all entries where the key is "unwanted":
	err = obj.DeleteElems(func(key []byte, i Iter) bool {
		return string(key) == "unwanted")
	}, nil)

	// Alternative version with prefiltered keys:
	err = obj.DeleteElems(nil, map[string]struct{}{"unwanted": {}})

(*Array).DeleteElems(fn func(i Iter) bool) will call back fn for each array value. If the function returns true the element is deleted in the array.

	// The array we are modifying
	var array *simdjson.Array

	// Delete all entries that are strings.
	array.DeleteElems(func(i Iter) bool {
		return i.Type() == TypeString
	})

Serializing parsed json

It is possible to serialize parsed JSON for more compact storage and faster load time.

To create a new serialized use NewSerializer. This serializer can be reused for several JSON blocks.

The serializer will provide string deduplication and compression of elements. This can be finetuned using the CompressMode setting.

To serialize a block of parsed data use the Serialize method.

To read back use the Deserialize method. For deserializing the compression mode does not need to match since it is read from the stream.

Speed and size on two documents from the test corpus, twitter (mostly strings) and canada (mostly floats), medians of three one-second runs on the machine named under Performance vs encoding/json. Reproduce with go test -run '^$' -bench 'BenchmarkSerialize$|BenchmarkDeSerialize$':

Compress Mode twitter: % of JSON serialize deserialize canada: % of JSON serialize deserialize
None 69.9% 3211 MB/s 14777 MB/s 69.3% 4804 MB/s 6789 MB/s
Fast 11.0% 1986 MB/s 8625 MB/s 36.2% 1586 MB/s 2887 MB/s
Default 10.6% 1571 MB/s 8645 MB/s 29.7% 770 MB/s 2067 MB/s
Best 7.3% 1042 MB/s 7947 MB/s 24.2% 501 MB/s 1822 MB/s

String deduplication is what makes the difference between the two documents: twitter repeats a modest vocabulary and compresses to a tenth, while canada is two million distinct coordinates and has nothing to deduplicate. The earlier version of this table was measured on parking-citations-1M, which the asset host no longer serves, so it could not be reproduced.

Performance vs encoding/json

Parsing the benchmark corpus, against encoding/json unmarshalling the same documents into an interface{}. Both columns are medians of six one-second runs, taken with the parser warm - Parse reusing a ParsedJson, which is how a service would call it.

document encoding/json simdjson, copy simdjson, nocopy faster by
apache_builds 694 µs, 183 MB/s 57 µs, 2225 MB/s 56 µs, 2277 MB/s 12.1x
canada 16.00 ms, 141 MB/s 2.89 ms, 778 MB/s 2.89 ms, 778 MB/s 5.5x
citm_catalog 9.03 ms, 191 MB/s 479 µs, 3603 MB/s 471 µs, 3670 MB/s 18.8x
github_events 339 µs, 192 MB/s 25 µs, 2614 MB/s 25 µs, 2582 MB/s 13.6x
gsoc-2018 12.36 ms, 269 MB/s 445 µs, 7476 MB/s 527 µs, 6317 MB/s 27.7x
instruments 1.39 ms, 158 MB/s 121 µs, 1825 MB/s 120 µs, 1833 MB/s 11.5x
marine_ik 25.45 ms, 117 MB/s 3.21 ms, 928 MB/s 3.22 ms, 926 MB/s 7.9x
mesh 6.30 ms, 115 MB/s 1.01 ms, 719 MB/s 1.01 ms, 718 MB/s 6.3x
mesh.pretty 9.21 ms, 171 MB/s 1.13 ms, 1393 MB/s 1.13 ms, 1394 MB/s 8.1x
numbers 1.02 ms, 147 MB/s 216 µs, 694 MB/s 217 µs, 693 MB/s 4.7x
random 3.62 ms, 141 MB/s 323 µs, 1580 MB/s 316 µs, 1616 MB/s 11.2x
twitter 3.79 ms, 167 MB/s 221 µs, 2863 MB/s 224 µs, 2824 MB/s 17.2x
twitterescaped 3.56 ms, 158 MB/s 336 µs, 1676 MB/s 459 µs, 1225 MB/s 10.6x
update-center 3.59 ms, 149 MB/s 268 µs, 1993 MB/s 266 µs, 2004 MB/s 13.4x

Geometric mean 10.8x, from 4.7x on an array of numbers to 27.7x on a large string-heavy document. The two ends of that range are the honest summary: the more of a document is strings and structure, the more there is for the vector code to do, and numbers is almost entirely strconv-shaped work that both libraries have to do byte by byte.

Allocation is the other half, and the gap is larger:

document encoding/json simdjson, copy
apache_builds 0.46 MB, 9719 allocs 31 B, 1 alloc
canada 10.58 MB, 392514 allocs 7.5 kB, 3 allocs
citm_catalog 5.13 MB, 95864 allocs 1.2 kB, 3 allocs
github_events 0.18 MB, 3365 allocs 21 B, 1 alloc
gsoc-2018 7.03 MB, 58605 allocs 3.1 kB, 3 allocs
instruments 0.87 MB, 13709 allocs 62 B, 1 alloc
marine_ik 21.88 MB, 614774 allocs 38 kB, 3 allocs
mesh 6.56 MB, 149495 allocs 3.0 kB, 3 allocs
mesh.pretty 6.71 MB, 149495 allocs 2.2 kB, 3 allocs
numbers 0.91 MB, 20023 allocs 81 B, 1 alloc
random 2.81 MB, 68019 allocs 484 B, 3 allocs
twitter 2.08 MB, 32124 allocs 248 B, 3 allocs
twitterescaped 2.26 MB, 32620 allocs 502 B, 3 allocs
update-center 2.71 MB, 50810 allocs 251 B, 3 allocs

encoding/json allocates a Go value per element; simdjson-go appends to buffers a reused ParsedJson already owns. On a cold parse - a fresh ParsedJson every time - it allocates those buffers instead, which for these documents is between 1 MB and 14 MB.

What this comparison is not. Parse builds a tape and stops; Unmarshal materialises every element as a Go value. Reading the tape afterwards costs more, and a caller who needs all of the document as Go values will not see 10x. benchmarks/BenchmarkMaterialise asks both sides for the same thing, and there the same corpus comes out at 2.7x with sonic roughly level with simdjson-go; benchmarks/README.md lists every asymmetry between the libraries and which of them were equalised. Compare within one group, never across.

How these were taken. Go 1.26.0, linux/amd64, AMD Ryzen AI MAX+ 395, both columns pinned to two physical cores that share an L3 slice. To reproduce:

# simdjson-go, both copy modes, from the repository root
taskset -c 2,3 go test -run '^$' -bench BenchmarkParseCorpus -benchtime 1s -count 6

# encoding/json and sonic, from benchmarks/
taskset -c 2,3 go test -run '^$' -bench 'BenchmarkEncodingJson|BenchmarkSonic' \
    -benchtime 1s -count 6

Numbers from a different machine will differ; the ratios travel better than the absolutes, and the shape - which documents gain most - travels best of all.

Inplace strings

The best performance is obtained by keeping the JSON message fully mapped in memory and using the WithCopyStrings(false) option. This prevents duplicate copies of string values being made but mandates that the original JSON buffer is kept alive until the ParsedJson object is no longer needed (ie iteration over the tape format has been completed).

In case the JSON message buffer is freed earlier (or for streaming use cases where memory is reused) WithCopyStrings(true) should be used (which is the default behaviour).

The two are much closer than they used to be. Copying a string no longer means walking it a second time - the copy and the search for the closing quote are one pass, and a string with no escapes is copied with append rather than word by word - so the difference is now the memory traffic of the copy itself:

document copy nocopy difference
apache_builds 2225 MB/s 2277 MB/s -2%
citm_catalog 3603 MB/s 3670 MB/s -2%
github_events 2614 MB/s 2582 MB/s +1%
gsoc-2018 7476 MB/s 6317 MB/s +18%
twitter 2863 MB/s 2824 MB/s +1%
twitterescaped 1676 MB/s 1225 MB/s +37%
update-center 1993 MB/s 1993 MB/s 0%
canada, mesh, numbers ~equal ~equal 0%

Two of those want explaining, since they say copying is faster: twitterescaped and gsoc-2018 are the documents where nearly every string carries escapes, and with WithCopyStrings(false) such a string still has to be decoded somewhere - it goes to the string buffer anyway - while the branch that avoids the copy entirely never fires. What the option really saves is the copy of strings that need no decoding, which is why the documents with plain strings show a couple of percent and the ones with none at all (canada, mesh, numbers) show nothing.

So WithCopyStrings(false) is worth reaching for when the message will outlive the parse anyway. It is not worth restructuring a service around.

Design

simdjson-go follows the same two stage design as simdjson. During the first stage the structural elements ({, }, [, ], :, and ,) are detected and forwarded as offsets in the message buffer to the second stage. The second stage builds a tape format of the structure of the JSON document.

Note that in contrast to simdjson, simdjson-go outputs uint32 increments (as opposed to absolute values) to the second stage. This allows arbitrarily large JSON files to be parsed (as long as a single (string) element does not surpass 4 GB...).

Also, for better performance, both stages run concurrently as separate go routines and a go channel is used to communicate between the two stages.

Stage 1

Stage 1 was originally converted from C code containing the SIMD intrinsics to Golang assembly using c2goasm. That C code is no longer part of this repository, and the assembly has been maintained by hand since - the AVX-512 variants in particular were written directly - so the .s files, not any generator, are the definition of what stage 1 does. It essentially consists of five separate steps, being:

  • find_odd_backslash_sequences: detect backslash characters used to escape quotes
  • find_quote_mask_and_bits: generate a mask with bits turned on for characters between quotes
  • find_whitespace_and_structurals: generate a mask for whitespace plus a mask for the structural characters
  • finalize_structurals: combine the masks computed above into a final mask where each active bit represents the position of a structural character in the input message.
  • flatten_bits_incremental: output the active bits in the final mask as incremental offsets.

For more details you can take a look at the various test cases in find_subroutines_amd64_test.go to see how the individual routines can be invoked (typically with a 64 byte input buffer that generates one or more 64-bit masks).

There is one final routine, find_structural_bits_in_slice, that ties it all together and is invoked with a slice of the message buffer in order to find the incremental offsets.

Stage 2

During Stage 2 the tape structure is constructed. It is essentially a single function that jumps around as it finds the various structural characters and builds the hierarchy of the JSON document that it processes. The values of the JSON elements such as strings, integers, booleans etc. are parsed and written to the tape.

Any errors (such as an array not being closed or a missing closing brace) are detected and reported back as errors to the client.

Tape format

Similarly to simdjson, simdjson-go parses the structure onto a 'tape' format. With this format it is possible to skip over arrays and (sub)objects as the sizes are recorded in the tape.

simdjson-go format is exactly the same as the simdjson tape format with the following 2 exceptions:

  • In order to support ndjson, it is possible to have more than one root element on the tape. Also, to allow for fast navigation over root elements, a root points to the next root element (and as such the last root element points 1 index past the length of the tape).

A "NOP" tag is added. The value contains the number of tape entries to skip forward for next tag.

  • Strings are handled differently, unlike simdjson the string size is not prepended in the String buffer but is added as an additional element to the tape itself (much like integers and floats).
    • In case WithCopyStrings(false) Only strings that contain special characters are copied to the String buffer in which case the payload from the tape is the offset into the String buffer. For string values without special characters the tape's payload points directly into the message buffer.
    • In case WithCopyStrings(true) (default): Strings are always copied to the String buffer.

For more information, see TestStage2BuildTape in stage2_build_tape_test.go.

Fuzz Tests

simdjson-go has been extensively fuzz tested to ensure that input cannot generate crashes and that output matches the standard library.

The fuzz tests are included as Go 1.18+ compatible tests.

License

simdjson-go is released under the Apache License v2.0. You can find the complete text in the file LICENSE.

Contributing

Contributions are welcome, please send PRs for any enhancements.

If your PR include parsing changes please run fuzz testers for a couple of hours.

Documentation

Index

Examples

Constants

View Source 
const (
	TagString      = Tag('"')
	TagInteger     = Tag('l')
	TagUint        = Tag('u')
	TagFloat       = Tag('d')
	TagNull        = Tag('n')
	TagBoolTrue    = Tag('t')
	TagBoolFalse   = Tag('f')
	TagObjectStart = Tag('{')
	TagObjectEnd   = Tag('}')
	TagArrayStart  = Tag('[')
	TagArrayEnd    = Tag(']')
	TagRoot        = Tag('r')
	TagNop         = Tag('N')
	TagEnd         = Tag(0)
)
View Source 
const JSONTAGMASK = 0xff << JSONTAGOFFSET
View Source 
const JSONTAGOFFSET = 56
View Source 
const JSONVALUEMASK = 0xff_ffff_ffff_ffff
View Source 
const STRINGBUFBIT = 0x80_0000_0000_0000
View Source 
const STRINGBUFMASK = 0x7fffffffffffff

Variables

View Source 
var ErrPathNotFound = errors.New("path not found")

ErrPathNotFound is returned

View Source 
var TagToType = [256]Type{
	TagString:      TypeString,
	TagInteger:     TypeInt,
	TagUint:        TypeUint,
	TagFloat:       TypeFloat,
	TagNull:        TypeNull,
	TagBoolTrue:    TypeBool,
	TagBoolFalse:   TypeBool,
	TagObjectStart: TypeObject,
	TagArrayStart:  TypeArray,
	TagRoot:        TypeRoot,
}

TagToType converts a tag to type. For arrays and objects only the start tag will return types. All non-existing tags returns TypeNone.

Functions

func ParseNDStream 

func ParseNDStream(r io.Reader, res chan<- Stream, reuse <-chan *ParsedJson)

ParseNDStream will parse a stream and return parsed JSON to the supplied result channel. The method will return immediately. Each element is contained within a root tag. 

<root>Element 1</root><root>Element 2</root>...

Each result will contain an unspecified number of full elements, so it can be assumed that each result starts and ends with a root tag. The parser will keep parsing until writes to the result stream blocks. A stream is finished when a non-nil Error is returned. If the stream was parsed until the end the Error value will be io.EOF The channel will be closed after an error has been returned. An optional channel for returning consumed results can be provided. There is no guarantee that elements will be consumed, so always use non-blocking writes to the reuse channel.

func SupportedCPU 

func SupportedCPU() bool

SupportedCPU will return whether the CPU is supported.

On amd64 this is still the AVX2 and CLMUL test it always was. The portable implementation added alongside the assembly is not reachable from this build: the two string routines are the hottest thing in stage 2, and giving them a runtime feature test to load and branch on measured at over 2% on string heavy documents - a real cost to every user, paid so that a pre-2013 CPU need not pass a build tag.

A CPU that lands here can have the portable implementation by building with -tags=noasm, which is what the error from Parse says. Every platform that has no assembly at all gets it without asking.

Types

type Array 

type Array struct {
	// contains filtered or unexported fields
}

Array represents a JSON array. There are methods that allows to get full arrays if the value type is the same. Otherwise an iterator can be retrieved.

Example 
if !SupportedCPU() {
	// Fake it
	fmt.Println("Found array\nType: int value: 116\nType: int value: 943\nType: int value: 234\nType: int value: 38793")
	return
}
input := `{
    "Image":
    {
        "Animated": false,
        "Height": 600,
        "IDs":
        [
            116,
            943,
            234,
            38793
        ],
        "Thumbnail":
        {
            "Height": 125,
            "Url": "http://www.example.com/image/481989943",
            "Width": 100
        },
        "Title": "View from 15th Floor",
        "Width": 800
    },
	"Alt": "Image of city" 
}`
pj, err := Parse([]byte(input), nil)
if err != nil {
	log.Fatal(err)
}
i := pj.Iter()
i.AdvanceInto()

// Grab root
_, root, err := i.Root(nil)
if err != nil {
	log.Fatal(err)
}
// Grab top object
obj, err := root.Object(nil)
if err != nil {
	log.Fatal(err)
}

// Find element in path.
elem, err := obj.FindPath(nil, "Image", "IDs")
if err != nil {
	log.Fatal(err)
}
fmt.Println("Found", elem.Type)
if elem.Type == TypeArray {
	array, err := elem.Iter.Array(nil)
	if err != nil {
		log.Fatal(err)
	}
	array.ForEach(func(i Iter) {
		asString, _ := i.StringCvt()
		fmt.Println("Type:", i.Type(), "value:", asString)
	})
}

Output:
Found array
Type: int value: 116
Type: int value: 943
Type: int value: 234
Type: int value: 38793

func (*Array) AsFloat 

func (a *Array) AsFloat() ([]float64, error)

AsFloat returns the array values as float. Integers are automatically converted to float.

func (*Array) AsInteger 

func (a *Array) AsInteger() ([]int64, error)

AsInteger returns the array values as int64 values. Uints/Floats are automatically converted to int64 if they fit within the range.

func (*Array) AsString 

func (a *Array) AsString() ([]string, error)

AsString returns the array values as a slice of strings. No conversion is done.

func (*Array) AsStringCvt 

func (a *Array) AsStringCvt() ([]string, error)

AsStringCvt returns the array values as a slice of strings. Scalar types are converted. Root, Object and Arrays are not supported an will return an error if found.

func (*Array) AsUint64 

func (a *Array) AsUint64() ([]uint64, error)

AsUint64 returns the array values as float. Uints/Floats are automatically converted to uint64 if they fit within the range.

func (*Array) DeleteElems 

func (a *Array) DeleteElems(fn func(i Iter) bool)

DeleteElems calls the provided function for every element. If the function returns true the element is deleted in the array.

Example 
if !SupportedCPU() {
	// Fake it
	fmt.Println("Found array\nModified: {\"Image\":{\"Animated\":false,\"Height\":600,\"IDs\":[943,38793]},\"Alt\":\"Image of city\"}")
	return
}
input := `{
    "Image":
    {
        "Animated": false,
        "Height": 600,
        "IDs":
        [
            116,
            943,
            234,
            38793
        ]
    },
	"Alt": "Image of city" 
}`
pj, err := Parse([]byte(input), nil)
if err != nil {
	log.Fatal(err)
}
i := pj.Iter()
i.AdvanceInto()

// Grab root
_, root, err := i.Root(nil)
if err != nil {
	log.Fatal(err)
}
// Grab top object
obj, err := root.Object(nil)
if err != nil {
	log.Fatal(err)
}

// Find element in path.
elem, err := obj.FindPath(nil, "Image", "IDs")
if err != nil {
	log.Fatal(err)
}
fmt.Println("Found", elem.Type)
if elem.Type == TypeArray {
	array, err := elem.Iter.Array(nil)
	if err != nil {
		log.Fatal(err)
	}
	// Delete all integer elements that are < 500
	array.DeleteElems(func(i Iter) bool {
		if id, err := i.Int(); err == nil {
			return id < 500
		}
		return false
	})
}
b, err := root.MarshalJSON()
if err != nil {
	log.Fatal(err)
}
fmt.Println("Modified:", string(b))

Output:
Found array
Modified: {"Image":{"Animated":false,"Height":600,"IDs":[943,38793]},"Alt":"Image of city"}

func (*Array) FirstType 

func (a *Array) FirstType() Type

FirstType will return the type of the first element. If there are no elements, TypeNone is returned.

func (*Array) ForEach 

func (a *Array) ForEach(fn func(i Iter))

ForEach calls the provided function for every element.

func (*Array) Interface 

func (a *Array) Interface() ([]interface{}, error)

Interface returns the array as a slice of interfaces. See Iter.Interface() for a reference on value types.

func (*Array) Iter 

func (a *Array) Iter() Iter

Iter returns the array as an iterator. This can be used for parsing mixed content arrays. The first value is ready with a call to Advance. Calling after last element should have TypeNone.

func (*Array) MarshalJSON 

func (a *Array) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (*Array) MarshalJSONBuffer 

func (a *Array) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal all elements. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

type CompressMode 

type CompressMode uint8
const (
	// CompressNone no compression whatsoever.
	CompressNone CompressMode = iota

	// CompressFast will apply light compression,
	// but will not deduplicate strings which may affect deserialization speed.
	CompressFast

	// CompressDefault applies light compression and deduplicates strings.
	CompressDefault

	// CompressBest
	CompressBest
)

type Element 

type Element struct {
	// Name of the element
	Name string
	// Type of the element
	Type Type
	// Iter containing the element
	Iter Iter
}

Element represents an element in an object.

type Elements 

type Elements struct {
	Elements []Element
	Index    map[string]int
}

Elements contains all elements in an object kept in original order. And index contains lookup for object keys.

func (Elements) Lookup 

func (e Elements) Lookup(key string) *Element

Lookup a key in elements and return the element. Returns nil if key doesn't exist. Keys are case sensitive.

func (Elements) MarshalJSON 

func (e Elements) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (Elements) MarshalJSONBuffer 

func (e Elements) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal all elements. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

type FloatFlag 

type FloatFlag uint64

FloatFlag is a flag recorded when parsing floats. 

const (
	// FloatOverflowedInteger is set when number in JSON was in integer notation,
	// but under/overflowed both int64 and uint64 and therefore was parsed as float.
	FloatOverflowedInteger FloatFlag = 1 << iota
)

func (FloatFlag) Flags 

func (f FloatFlag) Flags(more ...FloatFlag) FloatFlags

Flags converts the flag to FloatFlags and optionally merges more flags.

type FloatFlags 

type FloatFlags uint64

FloatFlags are flags recorded when converting floats.

func (FloatFlags) Contains 

func (f FloatFlags) Contains(flag FloatFlag) bool

Contains returns whether f contains the specified flag.

type Iter 

type Iter struct {
	// contains filtered or unexported fields
}

Iter represents a section of JSON. To start iterating it, use Advance() or AdvanceIter() methods which will queue the first element. If an Iter is copied, the copy will be independent.

func (*Iter) Advance 

func (i *Iter) Advance() Type

Advance will read the type of the next element and queues up the value on the same level.

func (*Iter) AdvanceInto 

func (i *Iter) AdvanceInto() Tag

AdvanceInto will read the tag of the next element and move into and out of arrays , objects and root elements. This should only be used for strictly manual parsing.

func (*Iter) AdvanceIter 

func (i *Iter) AdvanceIter(dst *Iter) (Type, error)

AdvanceIter will read the type of the next element and return an iterator only containing the object. If dst and i are the same, both will contain the value inside.

func (*Iter) Array 

func (i *Iter) Array(dst *Array) (*Array, error)

Array will return the next element as an array. An optional destination can be given.

func (*Iter) Bool 

func (i *Iter) Bool() (bool, error)

Bool returns the bool value.

func (*Iter) FindElement 

func (i *Iter) FindElement(dst *Element, path ...string) (*Element, error)

FindElement allows searching for fields and objects by path from the iter and forward, moving into root and objects, but not arrays. For example "Image", "Url" will search the current root/object for an "Image" object and return the value of the "Url" element. ErrPathNotFound is returned if any part of the path cannot be found. If the tape contains an error it will be returned. The iter will *not* be advanced.

Example 
if !SupportedCPU() {
	// Fake it
	fmt.Println("int\n100 <nil>")
	return
}
input := `{
    "Image":
    {
        "Animated": false,
        "Height": 600,
        "IDs":
        [
            116,
            943,
            234,
            38793
        ],
        "Thumbnail":
        {
            "Height": 125,
            "Url": "http://www.example.com/image/481989943",
            "Width": 100
        },
        "Title": "View from 15th Floor",
        "Width": 800
    },
	"Alt": "Image of city" 
}`
pj, err := Parse([]byte(input), nil)
if err != nil {
	log.Fatal(err)
}
i := pj.Iter()

// Find element in path.
elem, err := i.FindElement(nil, "Image", "Thumbnail", "Width")
if err != nil {
	log.Fatal(err)
}

// Print result:
fmt.Println(elem.Type)
fmt.Println(elem.Iter.StringCvt())

Output:
int
100 <nil>

func (*Iter) Float 

func (i *Iter) Float() (float64, error)

Float returns the float value of the next element. Integers are automatically converted to float.

func (*Iter) FloatFlags 

func (i *Iter) FloatFlags() (float64, FloatFlags, error)

FloatFlags returns the float value of the next element. This will include flags from parsing. Integers are automatically converted to float.

func (*Iter) Int 

func (i *Iter) Int() (int64, error)

Int returns the integer value of the next element. Integers and floats within range are automatically converted.

func (*Iter) Interface 

func (i *Iter) Interface() (interface{}, error)

Interface returns the value as an interface. Objects are returned as map[string]interface{}. Arrays are returned as []interface{}. Float values are returned as float64. Integer values are returned as int64 or uint64. String values are returned as string. Boolean values are returned as bool. Null values are returned as nil. Root objects are returned as []interface{}.

func (*Iter) MarshalJSON 

func (i *Iter) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the entire remaining scope of the iterator.

func (*Iter) MarshalJSONBuffer 

func (i *Iter) MarshalJSONBuffer(dst []byte) ([]byte, error)

MarshalJSONBuffer will marshal the remaining scope of the iterator including the current value. An optional buffer can be provided for fewer allocations. Output will be appended to the destination.

func (*Iter) Object 

func (i *Iter) Object(dst *Object) (*Object, error)

Object will return the next element as an object. An optional destination can be given.

func (*Iter) PeekNext 

func (i *Iter) PeekNext() Type

PeekNext will return the next value type. Returns TypeNone if next ends iterator.

func (*Iter) PeekNextTag 

func (i *Iter) PeekNextTag() Tag

PeekNextTag will return the tag at the current offset. Will return TagEnd if at end of iterator.

func (*Iter) Root 

func (i *Iter) Root(dst *Iter) (Type, *Iter, error)

Root returns the object embedded in root as an iterator along with the type of the content of the first element of the iterator. An optional destination can be supplied to avoid allocations.

func (*Iter) SetBool 

func (i *Iter) SetBool(v bool) error

SetBool can change a bool or null type to bool with the specified value. Attempting to change other types will return an error.

func (*Iter) SetFloat 

func (i *Iter) SetFloat(v float64) error

SetFloat can change a float, int, uint or string with the specified value. Attempting to change other types will return an error.

func (*Iter) SetInt 

func (i *Iter) SetInt(v int64) error

SetInt can change a float, int, uint or string with the specified value. Attempting to change other types will return an error.

func (*Iter) SetNull 

func (i *Iter) SetNull() error

SetNull can change the following types to null: Bool, String, (Unsigned) Integer, Float, Objects and Arrays. Attempting to change other types will return an error.

func (*Iter) SetString 

func (i *Iter) SetString(v string) error

SetString can change a string, int, uint or float with the specified string. Attempting to change other types will return an error.

func (*Iter) SetStringBytes 

func (i *Iter) SetStringBytes(v []byte) error

SetStringBytes can change a string, int, uint or float with the specified string. Attempting to change other types will return an error. Sending nil will add an empty string.

func (*Iter) SetUInt 

func (i *Iter) SetUInt(v uint64) error

SetUInt can change a float, int, uint or string with the specified value. Attempting to change other types will return an error.

func (*Iter) String 

func (i *Iter) String() (string, error)

String() returns a string value.

func (*Iter) StringBytes 

func (i *Iter) StringBytes() ([]byte, error)

StringBytes returns a string as byte array.

func (*Iter) StringCvt 

func (i *Iter) StringCvt() (string, error)

StringCvt returns a string representation of the value. Root, Object and Arrays are not supported.

func (*Iter) Type 

func (i *Iter) Type() Type

Type returns the queued value type from the previous call to Advance.

func (*Iter) Uint 

func (i *Iter) Uint() (uint64, error)

Uint returns the unsigned integer value of the next element. Positive integers and floats within range are automatically converted.

type Object 

type Object struct {
	// contains filtered or unexported fields
}

Object represents a JSON object.

func (*Object) DeleteElems 

func (o *Object) DeleteElems(fn func(key []byte, i Iter) bool, onlyKeys map[string]struct{}) error

DeleteElems will call back fn for each key. If true is returned, the key+value is deleted. A key filter can be provided for optional filtering. If fn is nil all elements in onlyKeys will be deleted. If both are nil all elements are deleted.

func (*Object) FindKey 

func (o *Object) FindKey(key string, dst *Element) *Element

FindKey will return a single named element. An optional destination can be given. The method will return nil if the element cannot be found. This should only be used to locate a single key where the object is no longer needed. The object will not be advanced.

func (*Object) FindPath 

func (o *Object) FindPath(dst *Element, path ...string) (*Element, error)

FindPath allows searching for fields and objects by path. Separate each object name by /. For example `Image/Url` will search the current object for an "Image" object and return the value of the "Url" element. ErrPathNotFound is returned if any part of the path cannot be found. If the tape contains an error it will be returned. The object will not be advanced.

Example 
if !SupportedCPU() {
	// Fake it
	fmt.Println("string\nhttp://www.example.com/image/481989943 <nil>")
	return
}
input := `{
    "Image":
    {
        "Animated": false,
        "Height": 600,
        "IDs":
        [
            116,
            943,
            234,
            38793
        ],
        "Thumbnail":
        {
            "Height": 125,
            "Url": "http://www.example.com/image/481989943",
            "Width": 100
        },
        "Title": "View from 15th Floor",
        "Width": 800
    },
	"Alt": "Image of city" 
}`
pj, err := Parse([]byte(input), nil)
if err != nil {
	log.Fatal(err)
}
i := pj.Iter()
i.AdvanceInto()

// Grab root
_, root, err := i.Root(nil)
if err != nil {
	log.Fatal(err)
}
// Grab top object
obj, err := root.Object(nil)
if err != nil {
	log.Fatal(err)
}

// Find element in path.
elem, err := obj.FindPath(nil, "Image", "Thumbnail", "Url")
if err != nil {
	log.Fatal(err)
}

// Print result:
fmt.Println(elem.Type)
fmt.Println(elem.Iter.String())

Output:
string
http://www.example.com/image/481989943 <nil>

func (*Object) ForEach 

func (o *Object) ForEach(fn func(key []byte, i Iter), onlyKeys map[string]struct{}) error

ForEach will call back fn for each key. A key filter can be provided for optional filtering.

func (*Object) Map 

func (o *Object) Map(dst map[string]interface{}) (map[string]interface{}, error)

Map will unmarshal into a map[string]interface{} See Iter.Interface() for a reference on value types.

func (*Object) NextElement 

func (o *Object) NextElement(dst *Iter) (name string, t Type, err error)

NextElement sets dst to the next element and returns the name. TypeNone with nil error will be returned if there are no more elements.

func (*Object) NextElementBytes 

func (o *Object) NextElementBytes(dst *Iter) (name []byte, t Type, err error)

NextElementBytes sets dst to the next element and returns the name. TypeNone with nil error will be returned if there are no more elements. Contrary to NextElement this will not cause allocations.

func (*Object) Parse 

func (o *Object) Parse(dst *Elements) (*Elements, error)

Parse will return all elements and iterators. An optional destination can be given. The Object will be consumed.

type ParsedJson 

type ParsedJson struct {
	Message []byte
	Tape    []uint64
	Strings *TStrings
	// contains filtered or unexported fields
}

func Parse 

func Parse(b []byte, reuse *ParsedJson, opts ...ParserOption) (*ParsedJson, error)

Parse an object or array from a block of data and return the parsed JSON. An optional block of previously parsed json can be supplied to reduce allocations.

func ParseND 

func ParseND(b []byte, reuse *ParsedJson, opts ...ParserOption) (*ParsedJson, error)

ParseND will parse newline delimited JSON objects or arrays. An optional block of previously parsed json can be supplied to reduce allocations.

func (*ParsedJson) Clone 

func (pj *ParsedJson) Clone(dst *ParsedJson) *ParsedJson

Clone returns a deep clone of the ParsedJson. If a nil destination is sent a new will be created.

func (*ParsedJson) ForEach 

func (pj *ParsedJson) ForEach(fn func(i Iter) error) error

ForEach returns each line in NDJSON, or the top element in non-ndjson. This will usually be an object or an array. If the callback returns a non-nil error parsing stops and the errors is returned.

Example 
if !SupportedCPU() {
	// Fake results
	fmt.Println("Got iterator for type: object\nFound element: URL Type: string Value: http://example.com/example.gif")
	return
}

// Parse JSON:
pj, err := Parse([]byte(`{"Image":{"URL":"http://example.com/example.gif"}}`), nil)
if err != nil {
	log.Fatal(err)
}

// Create an element we can reuse.
var element *Element
err = pj.ForEach(func(i Iter) error {
	fmt.Println("Got iterator for type:", i.Type())
	element, err = i.FindElement(element, "Image", "URL")
	if err == nil {
		value, _ := element.Iter.StringCvt()
		fmt.Println("Found element:", element.Name, "Type:", element.Type, "Value:", value)
	}
	return nil
})
if err != nil {
	log.Fatal(err)
}

Output:
Got iterator for type: object
Found element: URL Type: string Value: http://example.com/example.gif

func (*ParsedJson) Iter 

func (pj *ParsedJson) Iter() Iter

Iter returns a new Iter.

func (*ParsedJson) Reset 

func (pj *ParsedJson) Reset()

type ParserOption 

type ParserOption func(pj *internalParsedJson) error

ParserOption is a parser option.

func WithCopyStrings 

func WithCopyStrings(b bool) ParserOption

WithCopyStrings will copy strings so they no longer reference the input. For enhanced performance, simdjson-go can point back into the original JSON buffer for strings, however this can lead to issues in streaming use cases scenarios, or scenarios in which the underlying JSON buffer is reused. So the default behaviour is to create copies of all strings (not just those transformed anyway for unicode escape characters) into the separate Strings buffer (at the expense of using more memory and less performance). Default: true - strings are copied.

func WithValidateUTF8 

func WithValidateUTF8(b bool) ParserOption

WithValidateUTF8 will reject input that is not valid UTF-8.

The parser otherwise checks everything else about a string - escapes, surrogate pairs, unescaped control characters - but passes malformed UTF-8 through untouched, so a caller that treats the result as text can end up with invalid sequences it never put there.

Validation is a separate pass over the message. On the benchmark corpus it costs about 0.5% of parse time on number-heavy input and up to 19% on input that is both string-dense and largely non-ASCII, since the check runs far faster over plain ASCII than over multibyte sequences.

Default: false - UTF-8 is not checked, which is how the parser has always behaved.

type Serializer 

type Serializer struct {
	// contains filtered or unexported fields
}

Serializer allows to serialize parsed json and read it back. A Serializer can be reused, but not used concurrently.

func NewSerializer 

func NewSerializer() *Serializer

NewSerializer will create and initialize a Serializer.

func (*Serializer) CompressMode 

func (s *Serializer) CompressMode(c CompressMode)

func (*Serializer) Deserialize 

func (s *Serializer) Deserialize(src []byte, dst *ParsedJson) (*ParsedJson, error)

Deserialize the content in src. Only basic sanity checks will be performed. Slight corruption will likely go through unnoticed. And optional destination can be provided.

func (*Serializer) Serialize 

func (s *Serializer) Serialize(dst []byte, pj ParsedJson) []byte

Serialize the data in pj and return the data. An optional destination can be provided.

type Stream 

type Stream struct {
	Value *ParsedJson
	Error error
}

A Stream is used to stream back results. Either Error or Value will be set on returned results.

type TStrings 

type TStrings struct {
	B []byte
}

type Tag 

type Tag uint8

Tag indicates the data type of a tape entry

func (Tag) String 

func (t Tag) String() string

func (Tag) Type 

func (t Tag) Type() Type

Type converts a tag to a type. Only basic types and array+object start match a type.

type Type 

type Type uint8

Type is a JSON value type. 

const (
	TypeNone Type = iota
	TypeNull
	TypeString
	TypeInt
	TypeUint
	TypeFloat
	TypeBool
	TypeObject
	TypeArray
	TypeRoot
)

func (Type) String 

func (t Type) String() string

String returns the type as a string.

Source Files

View all Source files

Directories

Path Synopsis
find command
openrtb command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.