simdjson

README

simdjson

JSON for Go that locates a document's entire structure in a few vector passes and navigates that index instead of the bytes. Built on simd.go.

It provides both a drop-in encoding/json surface — Marshal, Unmarshal, Decoder, Encoder, Valid, Compact, Indent — and direct access to the index, so a field can be read out of a document without decoding the rest of it.

No cgo. The kernels are generated once from C and shipped as committed assembly, so the same code runs on amd64, arm64, riscv64, s390x, ppc64le and loong64.

go get github.com/sebishogun/simdjson

doc, err := simdjson.Parse(data)
if err != nil {
	return err
}

name := doc.Get("user", "name").String()
age := doc.Get("user", "age").Int()

doc.Get("items").ForEach(func(v simdjson.Value) bool {
	total += v.Key("score").Float()
	return true
})

API

Parsing and navigation

Parse validates the document against JSON's grammar and returns a Doc. Scan builds the same index and identifies the root without the grammar descent. MustParse panics instead of returning an error. Parser reuses its index across documents.

From a Doc: Root, Get, Path, Unmarshal.

A Value is a cursor into the index. Get, Key, Index and Path move; Kind, Len and Exists describe; String, StringNoCopy, Int, Float, Bool, IsNull, Time and Raw read; Decode unmarshals a subtree into a Go value. Iteration is available as callbacks — ForEach, ForEachKey — or as range-over-func: All, Values, Keys, Members.

GetPath and GetMany read one or several dotted paths straight from a byte slice.

On gjson's own published fixture and rotated paths (bench/getpath_rows_test.go), the trade is: gjson and jsonparser answer a one-shot path on the 1.1 KB document in ~145 ns where GetPath costs 518 — they scan forward and stop, this indexes first, and GetPath's documented promise (a document that does not parse yields Invalid) is what that index buys. From the SECOND query on the same document, Parse + Doc.Path runs each path at 143 ns — level with gjson's every-query price — and on documents twitter-sized and up the index wins from the first query (83 µs vs 105, further down). One-shot small-document gets are gjson's; everything repeated or sizable is ours.

encoding/json drop-in

Marshal, MarshalIndent, Unmarshal, Valid, Compact, Indent, HTMLEscape, NewDecoder, NewEncoder, RawMessage, Marshaler, Unmarshaler, and encoding.TextMarshaler / encoding.TextUnmarshaler. Decoder supports UseNumber, DisallowUnknownFields and Token. The omitempty, omitzero and ,string struct tags are honoured. The error types are aliases of the standard library's, so existing type assertions continue to work.

MarshalTo appends to a caller-supplied buffer and MarshalWrite writes to an io.Writer. Options selects encoder behaviour explicitly — Options.EscapeHTML, Options.SortMapKeys, Options.ValidateStrings, Options.OmitZeroStructFields — and has the same three methods.

Generated encoders

The struct encoder compiled at run time walks a field table, and that walk costs about 11% on a real document against straight-line code. For types you own, tools/structgen emits the straight-line code at build time:

//go:generate go run github.com/sebishogun/simdjson/tools/structgen -types User,Status

The generated file registers itself via RegisterEncoder; Marshal then uses it for that type everywhere it appears — top level, struct fields, slice elements, map values. Output is byte-identical to the reflect path, which the generator's own differential asserts. It declines any type it cannot encode exactly — maps, pointers, interfaces, []byte, embedded fields, tag options, types with their own MarshalJSON — and a declined type keeps the reflect encoder. Nothing is compiled at run time.

Generated encoders can also be written by hand against the same seam: RegisterEncoder, with AppendString, AppendInt, AppendUint, AppendFloat and AppendBool as the primitives, and the same byte-for-byte contract.

Editing, streaming and files

SetPath, SetRawPath and DeletePath rewrite a document in place by byte range, without decoding it. Skip returns the extent of the value at the start of a slice. An edit validates the document and the replacement before it splices — a Set that produces something unparseable is worse than an error — and that is the whole difference from sjson, which validates neither: on a 631 KB document, one field edit runs at 1,990 MB/s here against sjson's 4,492 (bench/editing_rows_test.go). The contract costs a validation pass; the number is what it costs.

ForEachLine, ForEachLineReader and ForEachLineReaderParallel read newline-delimited JSON.

OpenFile maps a file and returns a MappedFile with Bytes, Doc and Close.

Parse or Scan

Parse checks every value against the grammar and rejects what encoding/json rejects. Use it for input from outside.

Scan builds the index and identifies the root, skipping the descent that proves the parts never read are well-formed. Malformed input then yields wrong answers rather than errors — nothing reads out of bounds and nothing panics, but the result carries no guarantee. Use it for bytes you produced.

Validation is the larger part of the cost, which is what makes the distinction worth having.

Parser reuses its index between documents: 318 B per parse against 1,008 KB, and 286 µs against 345 µs on a 230 KB document.

Performance

Eight samples per row, shuffled, one process per row, minimum of each, on an idle amd64 machine; every table below is confirmed by an independent second pass (worst row-to-row deviation 3.3%). The encoding/json columns are the v1 engine; Go 1.27 intends to make the much faster jsonv2 engine the default, and make bench-v2 measures against it — stdlib struct decode rises to 614–712 MB/s (native v2 API: 759–896), and every row here still holds, at roughly 2–3× instead of 8×. Every Competitors run in the same process on the same bytes; bench/ is the harness.

Parsing — a document in, a navigable and validated structure out:

	this	fastjson	minio
twitter, 0.63 MB	198 µs	239 µs	305 µs	1.21×
citm, 1.73 MB	548 µs	695 µs	655 µs	1.20×
canada, 2.25 MB	1,136 µs	1,822 µs	5,504 µs	1.60×

Scan on the same three documents is 50 / 194 / 331 µs. It is a different operation: it does not validate.

Validating, against sonic, the other library doing it with vector instructions:

	this	sonic	encoding/json
twitter	82 µs	173 µs	1,242 µs	2.10×
citm	285 µs	440 µs	3,166 µs	1.54×
canada	885 µs	986 µs	4,175 µs	1.11×

Into Go values, each corpus into its natural struct (bench/decode_rows_test.go, minimum of three):

Unmarshal → struct	this	goccy	sonic	encoding/json
twitter	311 µs	361 µs	414 µs	2,562 µs
canada	2.58 ms	6.1 ms	2.64 ms	14.5 ms
citm	1.12 ms	0.97 ms	1.47 ms	7.5 ms
2 MB []float64	2.00 ms	5.1 ms	2.22 ms	11.2 ms

The memory column, same rows (-benchmem, minimum of two):

bytes / allocations per op	this	goccy	sonic	encoding/json
twitter	177 KB / 14	701 KB / 103	753 KB / 182	194 KB / 1,410
canada	1.0 MB / 966	4.2 MB / 56,538	4.9 MB / 2,588	3.1 MB / 3,095
citm	276 KB / 4,871	2.0 MB / 12,565	2.2 MB / 15,344	373 KB / 6,430
2 MB []float64	2.1 MB / 31	4.2 MB / 107,555	3.6 MB / 62	4.1 MB / 31

Interned strings, pooled scratch and one-walk numbers add up: fourteen allocations decode twitter into structs. goccy's citm speed lead costs 7.4× the memory and 2.6× the allocations; on the any tables the same holds — ours runs every shape at 40–70% of sonic's bytes and a quarter to a half of its allocations.

canada is level with sonic — 2.5% apart, inside the noise floor — after the compiled-array, extent-float and one-pass work.

The field's own fixtures — the Small/Medium/Large payloads every Go JSON README descends from (ported verbatim from buger/jsonparser; outputs byte-agreed with encoding/json before any timing; stdlib-compatible configs only, which most published tables for these fixtures do not use):

ns/op, minimum of eight	ours	goccy	segmentio	sonic	jsoniter	stdlib
Unmarshal small (190 B)	416	201	359	424	390	1,330
Unmarshal medium (2.2 KB)	2,055	1,484	1,974	2,635	3,185	9,771
Unmarshal large (28 KB)	20,522	16,184	29,840	33,186	54,117	117,785
Marshal small	91	98	112	144	194	212
Marshal medium	168	110	126	176	229	245
Marshal large	1,263	1,305	1,575	1,391	2,732	3,438

goccy's scanner core owns this size class, and wrong.md holds the instruction-level decomposition of why (537 decode instructions per field here against its 680 for everything). Getting the small decode row from 1,042 ns and 5.2 KB of garbage per call to 416 ns and 323 B — past sonic — is what this table's first measurement bought; Marshal small is the generated-encoder row (tools/structgen), now the row's best, and Marshal large is level with goccy at 3.3%, ours in front. Everything else in the column beats every library except goccy at every size. citm is goccy's row, cut from 41% to 22% by the one-walk integer parse (segmentio's 1,388 MB/s now trails our 1,463): tiny objects of small integers, where a hand-tuned scanner pays less per token than this design's index amortizes. An index-free prototype measured 4–5× SLOWER on every corpus, and the entry in wrong.md has the numbers. jsoniter and segmentio trail everywhere else measured — 191 and 404 MB/s on canada, 213 and 484 on the []float64 — and both are in the harness under stdlib-compatible configurations.

Out of Go values:

	this	sonic	goccy	encoding/json
Marshal, a struct	60 µs	35 µs	97 µs	112 µs
Marshal, map[string]struct, 256 entries	24 µs	23 µs	41 µs	62 µs

A decoded document — map[string]any with everything under it — encodes across cores when the output is a quarter megabyte or more: element ranges of a large []any shard to workers and the results stitch in order, byte-identical to the serial encode. Re-measured after that change, two passes of five, worse of the minima:

Marshal, decoded, sorted keys	this	sonic	goccy	encoding/json
twitter	237 µs	829 µs	1,824 µs	2,217 µs
citm	340 µs	1,401 µs	2,651 µs	3,294 µs
canada	2,284 µs	4,437 µs	6,976 µs	7,758 µs

sonic leads the two struct rows. Both are string escaping: its quote.c reserves worst-case output space and writes escapes inline in one vector pass, where this package's kernel stops at each byte needing an escape and returns to Go to emit it. Escaping costs 15.0 µs here on top of a 35.0 µs base; sonic's 27 µs covers escaping and UTF-8 validation together.

sonic's two passes differed by 19% on the struct row, where every other number here agreed within 1.6%, so that cell is a range.

Configuration: sonic.ConfigStd throughout, which sorts map keys, escapes HTML and validates strings. sonic.Marshal does none of the three; thirty calls on the same map produce five different outputs. Both are in the harness, the second marked not comparable.

Decoded into any — map[string]any and []any out of every corpus shape, MB/s, best of two passes, decodes cross-checked before timing. This family was sonic's on all twelve shapes until the any path was cured of per-string and per-key unquote and numbers stopped paying an allocation per box (the float payloads live in a document slab, like decoded strings):

into any, MB/s	ours	sonic	goccy	stdlib
twitter	570	683	361	197
citm	808	712	434	207
canada	483	371	180	149
numbers	520	551	204	164
github_events	601	829	455	202
apache_builds	494	695	428	200
gsoc-2018	1,424	1,989	744	283
instruments	453	570	294	175
update-center	382	466	264	172
mesh	386	362	152	129
mesh.pretty	849	646	301	183
marine_ik	413	342	157	128

Bold marks a lead past the 8.3% noise floor; unmarked cells are statistically level, measured against sonic v1.15.2. The split follows the data's shape: this package holds the array-heavy corpora — canada 1.30×, citm, mesh.pretty, marine_ik — where its []any values carve exact-size from a document-scoped slab; sonic holds six string- and object-heavy shapes by 18–40%, where its assembled walker feeds map assignment faster than a compiled Go loop can. Two are level. If decoding into any is your hot path, the shape of your documents decides; measure both. goccy and stdlib trail throughout.

Text in, text out, against encoding/json, MB/s:

	twitter	citm	canada	vs stdlib
Valid	7,648	6,054	2,489	4.6–15.2×
Compact	1,588	2,154	2,164	4.2–5.5×
Indent	1,210	1,227	599	2.0–3.3×

Valid is at or ahead of sonic on all twelve corpus shapes — past the noise floor on eleven (2.1× on twitter, 2.5× on apache_builds, 1.9–2.2× across the small-document shapes) and statistically level on the twelfth. Three kernels carry it: stage one's quote parity by carry-less multiply (simd v1.11.0), the grammar walk fused into one scalar routine over the stage-one masks (simd v1.12.0), and — since simd v1.13.0 — the whole of Valid as a single fused pass, per-block masks that never leave registers feeding parity, escape validation and the grammar machine with no mask buffers written or read. That fusion is what closed gsoc-2018, 3.3 MB of escape-heavy strings and the last shape sonic held (1.43×, the measured price of the staged design): one pass now answers it 32% faster than the staged pipeline it replaces. A density probe routes number-dominated documents (canada is 94% number bytes) to the descent walk instead, which pays nothing per block between one number and the next; docs/wrong.md holds that measurement, alongside every rejected step on the way here.

Under concurrency — aggregate throughput, every goroutine decoding its own twitter into its own struct (the many-requests server shape; bench/parallel_curve_test.go):

MB/s aggregate	1 thread	4	16	32
ours	1,950	7,494	17,661	20,041
goccy	1,694	6,104	13,365	15,762
sonic	1,387	4,976	10,562	11,952
encoding/json	208	850	2,038	2,382

Fastest at every width, and the flattest curve is sonic's, not ours. This is the many documents axis; the one document axis — a single payload sharded across cores past 8 MB — is the at-scale family above, which no other library has at all.

Cold start — the first operation on a never-seen type, measured by building a fresh type per iteration (bench/coldstart_test.go), which is what a deploy's first request meets and what the Pretouch warm-up in published tables hides:

first contact, ns	ours	encoding/json	goccy	sonic
Unmarshal, 5-field struct	3,205	3,546	4,629	857,495

sonic's 268× is its JIT compiling the fresh type — the cost Pretouch warm-ups hide; ours is a table build, and structgen'd types pay nothing at all.

Streaming, 50,000 newline-delimited records, 6.5 MB:

	this	goccy	sonic	encoding/json
Decoder	9.7 ms	11.9 ms	13.7 ms	37.8 ms
Encoder	5.9 ms	6.9 ms	9.1 ms	9.6 ms

Allocation for the same input is 9.5 MB in 150,183 allocations, against goccy's 12.9 MB in 306,525.

Record size is the axis that decides it. Streams of two-kilobyte records -- real tweets, newline-delimited, decoded into any -- run 515 MB/s here against sonic's 505; at fifty-kilobyte records the per-value work is almost entirely the any-decode itself and sonic's assembled walker takes it, 505 against 445. The crossover is the same residual the any-decode table above prices, reached through a different door.

Small documents are the size where an index does not pay. It costs the same few passes whether the document is 64 bytes or a megabyte:

	this	fastjson	encoding/json
64 B	123 ns	40 ns	76 ns
200 B	276 ns	103 ns	233 ns
2 KB	890 ns	1,080 ns	3,437 ns
20 KB	7,951 ns	10,882 ns	24,271 ns

The crossover is between 200 bytes and 2 KB. Below it, encoding/json is the better choice.

Pulling one field out of a document. 10,000 items, 1.17 MB, one field read. Everything here validates the whole document:

	10,000 items
this — Parse	0.802 ms
valyala/fastjson	1.307 ms	1.63×
minio/simdjson-go	2.054 ms	2.56×
bytedance/sonic	5.344 ms	6.66×
goccy/go-json	8.794 ms	10.97×
encoding/json	9.764 ms	12.2×

fastjson led this table by 9% when it was first measured; the kernel work since put Parse 1.63× ahead. fastjson builds a value tree into a reusable arena rather than an index, so navigation afterwards is a pointer walk where this is a lookup into a position array.

Against lazy scanners. gjson and jsonparser scan for a path and stop at the first match rather than parsing the document. gjson.Get is not the same operation as Parse: it does not validate, and answers from input that is not JSON.

input	gjson.Get returns	valid JSON
{"a" 1} — no colon	"1"	no
{"a":1 — unterminated	"1"	no
{"a":01} — invalid number	"01"	no

With validation on both sides, on a 10,000-item document:

	gjson	this
both validating — gjson.Valid+Get against Parse+Get	739 µs	802 µs	gjson 1.08× — level
neither validating — gjson.Get against Scan+Get	0.1 µs	169 µs	gjson ~1,700×

For one field, stop-at-first-match wins by construction when nothing is validated, and validation brings the two level. Two comparisons where the operations do match:

reading the whole document once	time	result
gjson.Valid	732 µs	a bool
Scan	173 µs	a reusable index — 4.2×
Parse	809 µs	that index, and the grammar proved

gjson retains nothing, so each Get rescans from byte zero, while this indexes once. Reading items.N.score for N across a 10,000-item document:

queries	gjson	this
1	0.1 µs	169 µs	gjson ~1,700×
10	3.1 µs	170 µs	gjson 55×
100	220 µs	205 µs	1.07× — level
1,000	20.2 ms	3.1 ms	6.4×

The crossover is about a hundred queries per document. Both are quadratic — gjson rescans to reach element N, Index(j) walks j elements — but each step here is a lookup rather than a byte scan.

gjson offers a path language this does not: wildcards, #(age>45) filters, about fifteen modifiers, JSON Lines and custom modifiers. This has Get, Key, Index, Path and ForEach. Its own documentation states that the Get* functions "expect that the json is well-formed" and that bad JSON "may return back unexpected results", which is what the validating row above measures.

Choosing. gjson or jsonparser for pulling a field or two from a document you produced. This package when the document comes from outside and must be checked, when it will be queried more than a few hundred times, or when the target is not amd64.

At a gigabyte and beyond

Real documents repeated to size, with a deliberate best and worst case:

1 GB, one piece	Scan	Valid	Parse	encoding/json.Valid
best case: minified ASCII, long values	7,302 MB/s	5,859	5,143	583
worst case: nothing but brackets and escapes	1,166	1,497	707	471

Six times between the two for Scan. A single throughput number for a JSON parser is an average over shapes that differ by that much.

Those rows are single-threaded. From 8 MB up, Scan and Parse build the structural index across cores — segments are indexed in parallel and the bracket pairs that cross a segment are merged serially, with output bit-identical to the single-threaded path, errors included. 64 MB of numbers-heavy JSON scans at 31.1 GB/s on 32 cores against 5.1 single-threaded; BenchmarkParallelScan reproduces it. When the root is an array of containers — the shape huge documents have — the bracket index gives every element's exact extent, and the grammar walks themselves shard across workers, ranges balanced by bytes rather than element count so twenty-eight two-megabyte documents split as well as three million records. At 64 MB: Valid 2.2 → 12.2 GB/s, Parse 1.8 → 11.5 GB/s, and Unmarshal into a struct slice 1.95 → 15.3 GB/s — each held identical to its serial path by a differential, errors included; other shapes walk serially over the same index. Compact and Indent join them — validation through the same parallel walk, and each transform sharded two-phase off the masks with its two-value writer state (depth and the pending-newline flag) carried across segment folds — at 0.5 → 2.5 GB/s and 0.27 → 1.23 GB/s respectively on 60 MB documents.

Past 2 GiB, Parse and Scan return an error naming the alternative — see Limits. That alternative is Decoder, which has no size limit. Ten gigabytes of tweets, 2.93 M records of about 3.4 KB each, from cd bench && go test -run TestHuge -huge -huge-bytes 10000000000 . — better of two passes, worse of the two heap peaks:

10 GB	decoded into	time	throughput	peak heap
line-delimited	Value	3.46 s	2,892 MB/s	8.0 MB
line-delimited	struct, 4 fields	5.03 s	1,988	9.4 MB
line-delimited	map[string]any	30.03 s	333	9.3 MB
one array	Value	3.66 s	2,731	8.5 MB
one array	struct, 4 fields	5.04 s	1,985	8.9 MB
one array	map[string]any	29.82 s	335	9.2 MB
one array, 300 M small elements	struct, 3 fields	22.65 s	441	9.3 MB

Under ten megabytes of heap for ten gigabytes of input, because nothing is held whole.

The decode target sets the throughput more than the parser does. Value builds no Go value and is the parser's own rate; map[string]any allocates a map and an interface per field and is 8× slower on identical bytes. Each row names its target for that reason.

Whole-document Unmarshal of a root array eight megabytes and up decodes across cores: element extents come from the parallel index, workers decode straight into the result slice, and any anomaly falls back to the serial decode, which owns the error. 32 MB of tweets into structs runs at 15.3 GB/s against 1.95 single-threaded — 7.9×, minimum of three — with values and errors identical to the serial path by differential.

A single enormous array works as well as line-delimited records. Read the opening bracket with Token, then More and Decode, as with the standard library:

dec := simdjson.NewDecoder(r)
if _, err := dec.Token(); err != nil { // the opening [
	return err
}
for dec.More() {
	var rec Record
	if err := dec.Decode(&rec); err != nil {
		return err
	}
	process(rec)
}

An object works the same way, with Token for each key and Decode for the value after it.

Nine shapes

twitter, citm and canada cover strings-and-objects, objects-and-whitespace and numbers. shapes_test.go adds deep nesting, wide objects, long strings, escape-heavy strings, non-ASCII, bare numbers, bare literals, pretty-printed and empty containers — each about a megabyte, each checked against encoding/json through every entry point before it is timed. Scan holds 12.3–12.5 GB/s on all of them except the two that are nothing but brackets.

The shape of it

Every row in the tables above, drawn. Ratio is time ÷ this library's time on the same bytes; the dashed line is 1.0, so bars below it are rows another library wins — and they are all here, because a chart that only shows the winning side is a sales pitch. The throughput charts carry the same rows in MB/s. All figures are regenerable and honest by construction: the snapshot they are drawn from (docs/bench/) names the machine, the instruction-set tier, the Go version and the date, and make bench-all re-measures and re-renders.

Raw throughput, for the record:

How these were measured. Every benchmark runs in its own fresh process — no benchmark's warm cache, branch history or allocator state carries into the next — and the order is shuffled per run. Each number is the minimum of eight samples, the estimator this repository's gate uses (layout noise is one-sided; the minimum converges to the true code speed). The machine is quiet, the tier is the one named in the snapshot (simd.Tier()), and the rivals run in the same process family on the same bytes. Slow rows — a benchmark whose single iteration exceeds the discovery threshold — are skipped and listed in the snapshot rather than run for hours; -include-slow restores them. The full record is make bench-all; the raw gate numbers are in testdata/bench/.

Limits

Document size. Parse and Scan index a document in one piece and cap at 2 GiB, because a bracket position is an int32 and the index is already 0.93× the size of the document; int64 positions would take it past 1.4× and charge every ordinary parse for a size that has a better answer. Above the cap they return an error naming Decoder, which streams in 64 KiB buffers and has no limit.

Whole-document decoding. Reading an entire document into Go values is slower than the standard library's single fused decode. An index pays for reaching into a document, not for reading all of it.

Strings with escapes are not zero-copy. A string containing no backslash is returned without copying out of the document; one with an escape is decoded into a new string.

Small inputs. Below roughly a kilobyte the fixed cost of the index is the whole cost. See the table above.

Correctness

Correctness is defined as agreeing with encoding/json and tested that way: hand-written cases, 2,000 randomised documents built from atoms chosen to collide (structure inside strings, escaped quotes, escaped backslashes, surrogate pairs), and differential fuzzing.

Eight fuzz targets compare against the standard library — parse, unmarshal, marshal, text operations, Decoder, Token, streamed array elements and UTF-8 validation — and demand the same bytes and the same error-or-not, not merely the same meaning.

go test ./...
go test -run '^$' -fuzz FuzzAgainstStdlib -fuzztime 60s
make verify        # fmt, vet, tests, race, every instruction tier, purego
make fuzz          # every differential target

The suite runs against each instruction tier separately (scalar, sse2, avx2, avx512) and under -tags purego, so every dispatch path is covered rather than only the one the build machine selects.

Findings that shaped the implementation, including measurements that argued against changes that were then reverted, are recorded in docs/wrong.md.

How it works

Two stages, the design C++ simdjson introduced.

Stage one classifies the document with vector compares and answers everything that follows with bit arithmetic. Three passes produce a bitmask each — one bit per input byte — for the quotes, the backslashes and the six structural characters. A conventional parser reads a byte and branches on what it is, which is a dependent, unpredictable branch per byte. This has no per-byte branch at all.

Stage two walks the surviving positions. A megabyte document might hold fifty thousand structural characters, so the second stage sees fifty thousand items rather than a million bytes.

The difficulty is in stage one. A { inside a string is text, and a " preceded by an odd number of backslashes closes nothing — in "a\\" the quote follows two backslashes and does close the string, while in "a\" it follows one and does not. Both are resolved before any position is interpreted, as arithmetic over sixty-four bytes at a time:

• which quotes are escaped — adding the odd-length backslash-run starts back into the backslash mask propagates a carry through each run and lands it one past the run's end, turning "the parity of this run" into a single add;
• which bytes are inside a string — an inclusive prefix XOR of the surviving quote mask, six shift-and-xor steps per word, with the parity carried into the next word by sign-extending its top bit;
• which structural characters survive — an and-not.

None of it costs anything per match.

Streaming indexes per buffer rather than per value, in partial mode, which treats a value cut in half by the end of the buffer as a fact to report rather than an error: it indexes what is there and records how far that reaches. Array elements are batched by bytes rather than by count — an element of a megabyte fills a batch alone, a hundred-byte record shares one with six hundred others — and the batch boundary is read off the index rather than found by a separate scan. A sustained Value loop over records goes further: batches are capped so the buffer holds the next one, a background task prepares it — index, scan, and validation fanned across cores — while the current one drains, and delivery hands each record out from its staged extent alone, with no whitespace skip, bracket match or validate left on the mainline. 1,450 → 1,974 MB/s on 64 MB of newline-delimited records. Decode keeps its serial per-value walk deliberately: decoding element k starts from the caller's variable as element k−1 left it, so the results chain by contract.

When to use it, and when not to

Every claim below cites a table in this README or a file in this repository; none of it is asserted from goodwill.

Use this library when:

• Documents are a megabyte or more. Fastest Go parse and validate on every corpus measured (Performance tables above), and past 8 MB fastest, period — the parallel family has no counterpart in Go or C++ (docs/cpp-baseline.md).
• You serve many requests. Fastest aggregate decode at every thread count, 20 GB/s at 32 threads, with the flattest competitor curve being sonic's, not ours (concurrency table).
• Streams: NDJSON, logs, exports. The streaming tables, the 10 GB rows under ten megabytes of heap, and a Value loop that pipelines its batches across cores.
• Numbers dominate. canada-class struct decode level with sonic, plain []float64 ahead of it, the numbers corpus ahead — the one-walk integer/float parsers and slab boxing did this.
• You query one document more than once. The second dotted-path query costs what gjson pays for every query; from twitter-size up the index wins from the first (GetPath notes above).
• You decode into any. Four leads, five levels, three floor-adjacent across twelve shapes (the any table).
• Deploy posture matters. No cgo, no JIT, no runtime executable memory, same code on six architectures, the fastest cold start in the field (first-contact row — sonic pays 2.5× compiling), and the conformance suites (JSONTestSuite, jsonchecker, UTF-8 stress) run in the ordinary test pass with zero disagreements against encoding/json.
• You own your types and want the last drop. tools/structgen emits compile-time encoders: level with goccy on the field's small Marshal fixture, byte-identical output enforced.

Prefer something else when:

• Tiny one-shot parses dominate. Sub-2 KB single documents: goccy's scanner core leads decode (176 ns vs our 404 on the field's small fixture), stdlib is fine, and Go 1.27's jsonv2 narrows every library's margin there for free.
• Small-struct Marshal is the hot path. sonic's fused JIT writer holds ~2× (the decomposition in wrong.md says exactly why, and what it would cost to chase) — if its posture fits your deploy.
• Dense tiny-object decode. citm-class shapes: goccy leads by ~1.2× (same scanner-core wall, measured three ways).
• One-shot field-gets on small documents. gjson answers in 145 ns where we pay 518 for the index and the validity promise; the trade flips on size or repetition (GetPath notes).
• Indented output is the product. MarshalIndent trails goccy's fused indent encoder by 1.24×.

Known costs, stated: the index is real memory (roughly document-sized; Parser reuse amortizes it); performance numbers are measured on amd64 — the other five architectures run the same code and the same tests, but the tables are from one machine; and published comparisons elsewhere often use lossy configurations (unsorted keys, skipped validation, warmed JITs) that this harness deliberately refuses, so our numbers for competitors run lower than their READMEs and are the defensible ones.

Status

Feature-complete against encoding/json: the drop-in surface passes the stdlib's own decode, encode, stream and tag test files, vendored and run in CI, and every entry point is differentially fuzzed against it. Wall-clock numbers are measured on amd64 (tier and machine named in each snapshot); the simd package underneath is correctness-verified on six architectures under emulation and wall-clock-verified on amd64 and arm64 NEON.

The rest of the family

All built on simd.go, which generates its kernels once from C and ships them as committed assembly for nine instruction sets — so none needs cgo, and none is amd64-only.

simd.go	474 vector operations over slices, bytes and text. The kernels everything else is built from.
simdblas	A BLAS backend for gonum. One blas64.Use call and mat, stat and optimize run on it.
simdcsv	CSV reading on one vector scan per record.
simdvec	Embedding search whose whole index scan is one matrix-vector product.

License

MIT — see LICENSE. Depends on simd.go (MIT).

Documentation

Overview

Package simdjson parses JSON by finding the whole document's structure in a few vector passes, then walking that instead of the bytes.

It is built on [simd.go](https://github.com/sebishogun/simd), so it needs no cgo and runs the same way on amd64, arm64, riscv64, s390x, ppc64le and loong64 — unlike the existing Go ports of simdjson, which are amd64 with hand-written assembly.

doc, err := simdjson.Parse(data)
name := doc.Get("user", "name").String()
age  := doc.Get("user", "age").Int()

How it works

Two stages, which is the design simdjson introduced.

Stage one finds every structural character — the braces, brackets, colons and commas — in one vector pass each, and works out which quotes really open and close strings rather than being escaped. A conventional parser reads a byte and branches on what it is, which is a dependent and unpredictable branch per byte; this makes eight branch-free passes over the document instead, and eight passes with no branches beat one pass with a branch per byte.

Stage two walks those positions. A document of a megabyte might have fifty thousand structural characters, so the second stage sees fifty thousand items rather than a million bytes.

What it is for

Pulling a few values out of a document, which is most of what JSON is used for and the case encoding/json is worst at — it decodes everything to reach anything. Doc.Get navigates the index without decoding what it passes.

It is not a replacement for encoding/json. There is no struct unmarshalling, no tags, no interfaces, no streaming. If you want a Go value, use the standard library; if you want three fields out of a large payload, this is several times faster.

Example

The case this package is for: a few values out of a document, without decoding the rest of it.

package main

import (
	"fmt"

	"github.com/sebishogun/simdjson"
)

func main() {
	data := []byte(`{
		"user": {"name": "ada", "age": 36, "tags": ["math", "engines"]},
		"meta": {"page": 1}
	}`)

	doc, err := simdjson.Parse(data)
	if err != nil {
		fmt.Println("bad json:", err)
		return
	}

	fmt.Println(doc.Get("user", "name").String())
	fmt.Println(doc.Get("user", "age").Int())
	fmt.Println(doc.Get("user", "tags").Index(1).String())
}

Output:
ada
36
engines

Index

• Variables
• func AppendBool(dst []byte, v bool) []byte
• func AppendFloat(dst []byte, v float64, bits int) []byte
• func AppendInt(dst []byte, v int64) []byte
• func AppendString(dst []byte, s string, opts Options) []byte
• func AppendUint(dst []byte, v uint64) []byte
• func Compact(dst *bytes.Buffer, src []byte) error
• func DeletePath(data []byte, path string) ([]byte, error)
• func ForEachLine(data []byte, fn func(Value) bool) error
• func ForEachLineReader(r io.Reader, fn func(Value) bool) error
• func ForEachLineReaderParallel(r io.Reader, fn func(Value) bool) error
• func HTMLEscape(dst *bytes.Buffer, src []byte)
• func Indent(dst *bytes.Buffer, src []byte, prefix, indent string) error
• func Marshal(v any) ([]byte, error)
• func MarshalIndent(v any, prefix, indent string) ([]byte, error)
• func MarshalTo(dst []byte, v any) ([]byte, error)
• func MarshalWrite(w io.Writer, v any) error
• func RegisterEncoder[T any](fn AppendFunc)
• func SetPath(data []byte, path string, v any) ([]byte, error)
• func SetRawPath(data []byte, path string, raw []byte) ([]byte, error)
• func Skip(data []byte) (start, end int, ok bool)
• func Unmarshal(data []byte, v any) error
• func Valid(data []byte) bool
• type AppendFunc
• type Decoder
  • func NewDecoder(r io.Reader) *Decoder
  • func (d *Decoder) Buffered() io.Reader
  • func (d *Decoder) Decode(out any) error
  • func (d *Decoder) DisallowUnknownFields()
  • func (d *Decoder) InputOffset() int64
  • func (d *Decoder) More() bool
  • func (d *Decoder) Token() (Token, error)
  • func (d *Decoder) UseNumber()
  • func (d *Decoder) Value() (Value, error)
• type Delim
• type Doc
  • func MustParse(data []byte) *Doc
  • func Parse(data []byte) (*Doc, error)
  • func Scan(data []byte) (*Doc, error)
  • func (d *Doc) Get(path ...string) Value
  • func (d *Doc) Path(path string) Value
  • func (d *Doc) Root() Value
  • func (d *Doc) Unmarshal(v any) error
• type Encoder
  • func NewEncoder(w io.Writer) *Encoder
  • func (e *Encoder) Encode(v any) error
  • func (e *Encoder) Options(o Options)
  • func (e *Encoder) SetEscapeHTML(on bool)
  • func (e *Encoder) SetIndent(prefix, indent string)
• type InvalidUnmarshalError
• type Kind
  • func (k Kind) String() string
• type MappedFile
  • func OpenFile(path string, validate bool) (*MappedFile, error)
  • func (m *MappedFile) Bytes() []byte
  • func (m *MappedFile) Close() error
  • func (m *MappedFile) Doc() *Doc
• type Marshaler
• type MarshalerError
• type Options
  • func (o Options) Marshal(v any) ([]byte, error)
  • func (o Options) MarshalTo(dst []byte, v any) ([]byte, error)
  • func (o Options) MarshalWrite(w io.Writer, v any) error
• type Parser
  • func (p *Parser) Parse(data []byte) (*Doc, error)
  • func (p *Parser) Scan(data []byte) (*Doc, error)
• type RawMessage
• type SyntaxError
  • func (e *SyntaxError) Error() string
• type Token
• type UnmarshalTypeError
• type Unmarshaler
• type UnsupportedTypeError
• type UnsupportedValueError
• type Value
  • func GetMany(data []byte, paths ...[]string) []Value
  • func GetPath(data []byte, path string) Value
  • func (v Value) All() iter.Seq2[int, Value]
  • func (v Value) Bool() bool
  • func (v Value) Decode(out any) error
  • func (v Value) Exists() bool
  • func (v Value) Float() float64
  • func (v Value) ForEach(fn func(Value) bool)
  • func (v Value) ForEachKey(fn func(string, Value) bool)
  • func (v Value) Get(path ...string) Value
  • func (v Value) Index(n int) Value
  • func (v Value) Int() int64
  • func (v Value) IsNull() bool
  • func (v Value) Key(name string) Value
  • func (v Value) Keys() iter.Seq[string]
  • func (v Value) Kind() Kind
  • func (v Value) Len() int
  • func (v Value) Members() iter.Seq2[string, Value]
  • func (v Value) Path(path string) Value
  • func (v Value) Raw() []byte
  • func (v Value) String() string
  • func (v Value) StringNoCopy() string
  • func (v Value) Time() time.Time
  • func (v Value) Values() iter.Seq[Value]

Examples

• Package
• Parse (StringsHideStructure)
• Parser
• Value.Exists
• Value.ForEach
• Value.ForEachKey

Variables

var Fast = Options{SortMapKeys: true}

Fast gives up HTML escaping and UTF-8 validation.

Use it when the output is not going into a page and the strings are known to be valid UTF-8 — decoded from JSON, read from a UTF-8 database column, built from Go string literals. The output is identical to Std's for any input that meets those conditions, and differs for any that does not.

Map keys are still sorted. Not sorting them is a bigger change than a few percent: it makes the same value encode differently on successive calls, which is a different promise rather than a faster one. Set SortMapKeys to false deliberately if that is wanted.

var Std = Options{EscapeHTML: true, ValidateStrings: true, SortMapKeys: true}

Std matches encoding/json byte for byte. It is what the package-level Marshal uses.

Functions

func AppendBool

func AppendBool(dst []byte, v bool) []byte

AppendBool appends "true" or "false".

func AppendFloat

func AppendFloat(dst []byte, v float64, bits int) []byte

AppendFloat appends v in the shortest form that round-trips, as JSON requires. bits is 32 or 64.

func AppendInt

func AppendInt(dst []byte, v int64) []byte

AppendInt, AppendUint, AppendFloat and AppendBool are the rest of what a generated encoder needs for scalar fields, and are the same code Marshal uses for them.

func AppendString

func AppendString(dst []byte, s string, opts Options) []byte

AppendString writes s as a quoted JSON string under opts, which is what a generated encoder needs for a string field. It is the same code Marshal uses, exported so that generated code cannot drift from it.

func AppendUint

func AppendUint(dst []byte, v uint64) []byte

AppendUint appends v in decimal.

func Compact

func Compact(dst *bytes.Buffer, src []byte) error

Compact appends the JSON in src to dst with insignificant whitespace removed.

Whitespace inside a string is significant and is kept; everything between tokens goes. Invalid input is reported and dst is left as it was, which is what encoding/json.Compact does — it validates as it copies.

func DeletePath

func DeletePath(data []byte, path string) ([]byte, error)

DeletePath returns data with the value at path removed, along with its key if it had one and the separating comma if it needed one.

A path that does not exist is not an error: the document comes back unchanged, which is what "make sure this is not there" should do.

func ForEachLine

func ForEachLine(data []byte, fn func(Value) bool) error

ForEachLine calls fn for each JSON value in data.

Whitespace between values, including the newlines, is skipped, so this reads NDJSON and equally a file of values with no separators at all. Input that is not valid JSON stops the walk and is returned as an error carrying the offset.

fn returning false stops the walk without an error, the same way Value.ForEach does.

The Value passed to fn is only valid for the duration of the call: it points into a batch that is reused for the values after it, which is what keeps this to a fixed amount of memory however long the input is.

func ForEachLineReader

func ForEachLineReader(r io.Reader, fn func(Value) bool) error

ForEachLineReader is ForEachLine over a stream.

Memory is one batch plus the index over it, so a file larger than memory is fine: ten gigabytes of NDJSON goes through this in under twenty megabytes.

func ForEachLineReaderParallel

func ForEachLineReaderParallel(r io.Reader, fn func(Value) bool) error

ForEachLineReaderParallel is ForEachLineReader across several goroutines.

fn is called on the calling goroutine, once per record, in input order. The parallelism is in the indexing, not in the callback, so fn needs no locking and sees records in the order they appeared.

The Value passed to fn is valid for the duration of the call.

Memory is bounded by the number of workers times the chunk size, whatever the length of the input.

An error stops the reader, drains the workers and is returned; so does fn returning false, without an error. The records before an error in the same chunk are still delivered — they are still good, and dropping them would silently truncate at a chunk boundary. Errors carry the byte offset in the whole stream, not in the chunk they were found in.

func HTMLEscape

func HTMLEscape(dst *bytes.Buffer, src []byte)

HTMLEscape appends src to dst with <, >, & and the two Unicode line terminators replaced by their \u escapes, so the result can go inside a <script> tag without ending it.

It does not parse. In well-formed JSON none of those five can appear outside a string literal, so replacing them wherever they occur is the same thing as replacing them inside strings, and encoding/json.HTMLEscape makes the same bet. Nothing here validates, which is also what encoding/json does.

func Indent

func Indent(dst *bytes.Buffer, src []byte, prefix, indent string) error

Indent appends the JSON in src to dst, one element per line, each nested level prefixed by one more copy of indent and every line by prefix.

Byte-for-byte what encoding/json.Indent produces, including the space after a colon and the empty object written as {} rather than opened and closed on two lines.

func Marshal

func Marshal(v any) ([]byte, error)

Marshal returns the JSON encoding of v.

It is encoding/json.Marshal's contract — the same escaping, the same tag handling, the same sorted map keys, the same treatment of nil — produced by an encoder compiled once per type rather than by walking reflect for every field. Where the two could differ they are held together by a differential fuzz test rather than by inspection.

func MarshalIndent

func MarshalIndent(v any, prefix, indent string) ([]byte, error)

MarshalIndent is Marshal followed by Indent, minus the proof: the bytes between them are this package's own output, compact and valid by construction, so the grammar walk Indent runs over input from outside proves nothing here. The masks are still built -- the writer lays out strings and depth from them -- and the walk was a fifth of the total.

func MarshalTo

func MarshalTo(dst []byte, v any) ([]byte, error)

MarshalTo appends the JSON encoding of v to dst and returns the extended slice.

The shape a server wants: one buffer, reused across responses, so encoding a stream of payloads does not allocate a new one for each.

func MarshalWrite

func MarshalWrite(w io.Writer, v any) error

MarshalWrite writes the JSON encoding of v to w, matching encoding/json.

func RegisterEncoder

func RegisterEncoder[T any](fn AppendFunc)

RegisterEncoder installs fn as the encoder for T, for Marshal and everything built on it -- including T appearing as a struct field, a slice element or a map value.

It must be called before the first encode of a value containing T, which in practice means from an init function in the package that owns the generated code. Registering after a type has been encoded once has no effect: the compiled encoder for it is already cached, and so is every encoder that has already inlined a reference to it.

Registering the same type twice replaces the earlier registration.

func SetPath

func SetPath(data []byte, path string, v any) ([]byte, error)

SetPath returns data with the value at path replaced by v, encoding v with Marshal.

Missing structure is created: setting `a.b.c` on `{}` gives `{"a":{"b":{"c":...}}}`. A numeric component creates an array only if it is 0 or the path already leads to an array; sjson pads with nulls for a larger index and that is a footgun rather than a feature, so an index past the end of an existing array appends instead.

The path grammar is Value.Path's, minus the wildcards: `*` and `?` have no single answer to write to, so a path containing either is an error.

func SetRawPath

func SetRawPath(data []byte, path string, raw []byte) ([]byte, error)

SetRawPath is SetPath with the replacement given as JSON text rather than a Go value.

raw is validated before it is spliced in, because a Set that produces a document which no longer parses is worse than an error.

func Skip

func Skip(data []byte) (start, end int, ok bool)

Skip returns the extent of the first JSON value in data: the offset of its first byte and the offset one past its last.

Validate and locate in one call, which sonic exposes as decoder.Skip. It is the operation behind "give me this value's bytes without decoding it" — a router picking one field out of a body, a proxy forwarding a subtree, a test comparing raw JSON.

ok is false if data holds no complete valid value. Trailing bytes after the first value are not an error and not included: `{} garbage` gives 0, 2, true.

func Unmarshal

func Unmarshal(data []byte, v any) error

func Valid

func Valid(data []byte) bool

Types

type AppendFunc

type AppendFunc func(dst []byte, p unsafe.Pointer, opts Options) []byte

AppendFunc writes v as JSON to the end of dst and returns the extended buffer. p points at a value of the registered type.

The contract is exact and unforgiving, because nothing checks it at run time: the bytes written must be the bytes Marshal would have written for the same value under the same Options, including the escaping and the field order. A registered encoder that disagrees produces wrong output with no error, so whatever writes one owes a differential test against Marshal.

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

A Decoder reads JSON values from a stream, one call to Decode per value.

Values may be separated by whitespace or by nothing at all; newline-delimited JSON is the case where they are separated by exactly one newline, and needs no special handling here.

func NewDecoder

func NewDecoder(r io.Reader) *Decoder

NewDecoder returns a Decoder reading from r.

It may read more from r than it needs to answer a call to Decode; whatever it has read and not used is available from Decoder.Buffered.

func (*Decoder) Buffered

func (d *Decoder) Buffered() io.Reader

Buffered returns a reader over the bytes read from the underlying reader and not yet consumed by Decode.

func (*Decoder) Decode

func (d *Decoder) Decode(out any) error

Decode reads the next JSON value from the stream and stores it in v.

It returns io.EOF when the stream holds no further value, which is what ends a read loop.

func (*Decoder) DisallowUnknownFields

func (d *Decoder) DisallowUnknownFields()

DisallowUnknownFields makes Decode report an error when the input names a field the destination struct does not.

func (*Decoder) InputOffset

func (d *Decoder) InputOffset() int64

InputOffset returns the position in the stream just after the most recently decoded value.

func (*Decoder) More

func (d *Decoder) More() bool

More reports whether there is another element in the array or object being read, or another value in the stream.

func (*Decoder) Token

func (d *Decoder) Token() (Token, error)

Token returns the next syntactic element: a Delim for a bracket, or the value of a string, number, bool or null.

Object keys come back as strings, in the position they appear. Commas and colons are consumed and never returned, which is what makes the token stream the same shape as encoding/json's.

It returns io.EOF when the input is exhausted. Token and Decoder.Decode interleave: after Token has returned the opening bracket of an array, Decode reads the next element of it, which is the whole point.

func (*Decoder) UseNumber

func (d *Decoder) UseNumber()

UseNumber makes Decode store a number in an any as a Number -- the digits as they were written -- rather than a float64, which cannot hold all of them.

func (*Decoder) Value

func (d *Decoder) Value() (Value, error)

Value returns the next value in the stream without decoding it into a Go value.

The same framing as Decoder.Decode -- separators consumed, batches reused -- stopping one step earlier: the value is handed back as a Value pointing into the batch rather than copied into a destination. That is what makes reading line-delimited JSON cheap, because most records in a log are read to pull two fields out of them and decoding the other twenty is waste.

The value is validated before it is returned, unlike Scan, because a caller stepping through a stream is asking "is this a record" and the answer has to mean something. That costs about a third of the throughput and is not optional.

The returned Value is only valid until the next call. It points into a buffer this Decoder reuses, and reusing it is the whole reason a ten gigabyte stream fits in twenty megabytes.

It returns io.EOF when the input is exhausted.

type Delim

type Delim = json.Delim

A Delim is one of the four bracket characters.

type Doc

type Doc struct {
	// contains filtered or unexported fields
}

Doc is a parsed document. It holds the input and its structural index; no values are decoded until they are asked for.

func MustParse

func MustParse(data []byte) *Doc

MustParse is Parse for input already known to be valid. It panics if it is not.

For tests, for constants compiled into the program, and for the top of a function that has already validated its input. fastjson exposes the same thing and for the same reason: an error return that can never fire is noise at the call site.

func Parse

func Parse(data []byte) (*Doc, error)

Parse indexes data and validates its structure.

The returned Doc keeps data — it is not copied, and every string a Value yields points into it unless the string contains an escape.

Example (StringsHideStructure)

Structure inside a string is text, which is the whole difficulty of stage one and is handled before any of it is interpreted.

package main

import (
	"fmt"

	"github.com/sebishogun/simdjson"
)

func main() {
	doc, err := simdjson.Parse([]byte(`{"a":"},{\"b\":2},[","c":1}`))
	if err != nil {
		fmt.Println("err:", err)
		return
	}
	fmt.Printf("%q\n", doc.Get("a").String())
	fmt.Println(doc.Get("c").Int())
}

Output:
"},{\"b\":2},["
1

func Scan

func Scan(data []byte) (*Doc, error)

Scan indexes data without validating it.

Parse walks the whole document and checks every value against JSON's grammar, which is what makes it safe for input you did not produce — and it is most of the cost. If the goal is three fields out of a payload your own service just serialised, validating the other nine thousand is work nobody asked for.

Scan skips it. The structural index is still built, so navigation works exactly as it does after Parse; what is gone is the recursive descent that proves the parts you never look at are well-formed.

What that costs

Malformed input gives wrong answers rather than errors. A missing colon, a trailing comma, a number like 10., an invalid escape — all are accepted, and the values around them may come back wrong or absent instead of failing. The index itself is still consistent, so nothing reads out of bounds and nothing panics; the result is simply not to be trusted.

Two things are still checked, because the index cannot be built without them: every string is terminated, and quotes balance. A document that fails either is rejected here too.

Use Parse for anything from outside. Use Scan when you produced the bytes.

func (*Doc) Get

func (d *Doc) Get(path ...string) Value

Get walks a path of object keys and returns the value at the end.

A missing key, or a path that runs into a non-object, yields an Invalid Value rather than an error — chaining is the common case and an error at every step would be unusable. Check Value.Exists.

func (*Doc) Path

func (d *Doc) Path(path string) Value

Path returns the value at a dot-separated path from the document's root.

func (*Doc) Root

func (d *Doc) Root() Value

Root returns the document's top-level value.

func (*Doc) Unmarshal

func (d *Doc) Unmarshal(v any) error

Unmarshal decodes an already-parsed document into v.

Use this when the same bytes are decoded more than once, or when a document is navigated first and decoded after: the parse is the expensive half and this skips it.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

An Encoder writes JSON values to a stream, one call to Encode per value, each followed by a newline.

func NewEncoder

func NewEncoder(w io.Writer) *Encoder

NewEncoder returns an Encoder writing to w, matching encoding/json's defaults: HTML characters escaped, strings checked for valid UTF-8.

func (*Encoder) Encode

func (e *Encoder) Encode(v any) error

Encode writes the JSON encoding of v to the stream, followed by a newline.

func (*Encoder) Options

func (e *Encoder) Options(o Options)

Options sets the whole option set at once, which is how the non-validating mode is reached from a stream. See Options.

func (*Encoder) SetEscapeHTML

func (e *Encoder) SetEscapeHTML(on bool)

SetEscapeHTML controls whether <, > and & are escaped. It is on by default.

func (*Encoder) SetIndent

func (e *Encoder) SetIndent(prefix, indent string)

SetIndent makes Encode write each value the way Indent would. An empty indent turns it off.

type InvalidUnmarshalError

type InvalidUnmarshalError = json.InvalidUnmarshalError

InvalidUnmarshalError describes an invalid argument passed to Unmarshal — the argument must be a non-nil pointer.

type Kind

type Kind uint8

Kind is the type of a JSON value.

const (
	Invalid Kind = iota
	Null
	Bool
	Number
	String
	Array
	Object
)

func (Kind) String

func (k Kind) String() string

type MappedFile

type MappedFile struct {
	// contains filtered or unexported fields
}

A MappedFile is a JSON document mapped into memory.

Close must be called, and no Value taken from the document may be used afterwards: the bytes go away with the mapping, and reading them then is a segmentation fault rather than a Go panic. Value.String copies, so a string taken from it outlives Close; Value.StringNoCopy and Value.Raw do not.

func OpenFile

func OpenFile(path string, validate bool) (*MappedFile, error)

OpenFile maps path into memory and indexes it.

validate says whether to prove the whole document well-formed, which is the difference between Parse and Scan: validating a two gigabyte file costs about four times what indexing it does, and a caller pulling one field out of a log does not need it.

func (*MappedFile) Bytes

func (m *MappedFile) Bytes() []byte

Bytes returns the mapped file's contents. It is valid until MappedFile.Close, and writing to it will fault: the mapping is read-only.

func (*MappedFile) Close

func (m *MappedFile) Close() error

Close unmaps the file and closes it.

func (*MappedFile) Doc

func (m *MappedFile) Doc() *Doc

Doc returns the parsed document. It is valid until MappedFile.Close.

type Marshaler

type Marshaler = json.Marshaler

Marshaler is the interface implemented by types that can marshal themselves into valid JSON.

It is an alias for json.Marshaler.

type MarshalerError

type MarshalerError = json.MarshalerError

MarshalerError is returned when a type's own MarshalJSON or MarshalText method returns an error.

type Options

type Options struct {
	// EscapeHTML writes `<`, `>` and `&` as <, > and &, and
	// rewrites U+2028 and U+2029, so the output can be embedded in an HTML
	// document without becoming script. encoding/json does this by default and
	// so does this package.
	//
	// Turning it off is worth a few percent and is safe only if the output
	// never reaches a page. Note that some other libraries have it off by
	// default, which is worth knowing when comparing their numbers.
	EscapeHTML bool

	// ValidateStrings replaces bytes that are not valid UTF-8 with U+FFFD,
	// which is what encoding/json does. Off, they are written through as-is,
	// producing output that is not valid JSON if the input was not valid UTF-8.
	//
	// This is the expensive one — on a document of non-ASCII text, validation
	// is about a third of the encode — and the right choice when the strings
	// come from somewhere that already guarantees UTF-8.
	ValidateStrings bool

	// SortMapKeys writes a map's keys in order. encoding/json always does, so
	// this is on in [Std] and every byte-for-byte comparison depends on it.
	//
	// Off, keys come out in whatever order the map iterates, which Go
	// deliberately randomises — so the same map encodes differently on
	// successive calls. That is fine for a payload nobody diffs and fatal for
	// a cache key, an ETag or a signature. encoding/json/v2 makes it opt-in;
	// this keeps v1's default and lets you turn it off, which is the safer way
	// round.
	SortMapKeys bool

	// OmitZeroStructFields drops every struct field holding its type's zero
	// value, as though each carried `omitzero`.
	//
	// New in encoding/json/v2 as an option, and useful for the case the tag
	// cannot serve: a type from another package, or a struct being encoded for
	// a wire format that treats absent and zero the same.
	//
	// It follows `omitzero` and not `omitempty`: an empty slice and an empty
	// map are their zero value only when nil, and a type with its own IsZero
	// method is asked. A field with an explicit tag keeps whatever the tag
	// said.
	OmitZeroStructFields bool
}

Options selects what an encoder checks and escapes.

The defaults match encoding/json exactly, because a drop-in replacement that quietly produces different bytes is worse than a slow one. Everything here is a way to buy speed by giving something up, and each says what.

func (Options) Marshal

func (o Options) Marshal(v any) ([]byte, error)

Marshal returns the JSON encoding of v under these options.

func (Options) MarshalTo

func (o Options) MarshalTo(dst []byte, v any) ([]byte, error)

MarshalTo appends the JSON encoding of v to dst under these options.

func (Options) MarshalWrite

func (o Options) MarshalWrite(w io.Writer, v any) error

MarshalWrite writes the JSON encoding of v to w.

The shape encoding/json/v2 added as MarshalWrite: encode straight into the destination rather than building a []byte and handing it over. For a large value going to a socket or a file this is the difference between one buffer and two.

It is not Encoder.Encode: that appends a newline, because it is for writing a stream of values. This writes exactly the value.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser parses documents, reusing its index buffers between them.

A server handling many payloads should keep one per goroutine: Parse allocates a fresh index each time, and for a document of a few hundred kilobytes that index is several times the size of the document itself. A Parser reuses it, so the second and later documents allocate almost nothing.

A Parser is not safe for concurrent use.

Example

A Parser reuses its index between documents, which is what a server handling a stream of payloads wants. The Doc it returns is only valid until the next Parse on the same Parser.

package main

import (
	"fmt"

	"github.com/sebishogun/simdjson"
)

func main() {
	var p simdjson.Parser

	for _, payload := range [][]byte{
		[]byte(`{"id":1}`),
		[]byte(`{"id":2}`),
	} {
		doc, err := p.Parse(payload)
		if err != nil {
			return
		}
		fmt.Println(doc.Get("id").Int())
	}
}

Output:
1
2

func (*Parser) Parse

func (p *Parser) Parse(data []byte) (*Doc, error)

Parse indexes and validates data, reusing p's buffers.

The returned Doc borrows those buffers, so it is only valid until the next call to Parse on the same Parser. Use Parse if a Doc has to outlive that.

func (*Parser) Scan

func (p *Parser) Scan(data []byte) (*Doc, error)

Scan indexes data without validating it, reusing p's buffers.

See Scan for what is given up, and Parser.Parse for the lifetime of the returned Doc.

type RawMessage

type RawMessage = json.RawMessage

RawMessage is a raw encoded JSON value. It implements json.Marshaler and json.Unmarshaler and can be used to delay JSON decoding or precompute a JSON encoding.

It is an alias for json.RawMessage, so the two are the same type.

type SyntaxError

type SyntaxError struct {
	Offset int64
	// contains filtered or unexported fields
}

A SyntaxError reports that the input is not valid JSON.

Offset is the byte in the input where the problem was found, which is the whole reason this is a type and not a string: the position used to be formatted into the message and a caller who wanted it had to parse English back out. encoding/json, encoding/json/v2, goccy and sonic all carry it; fastjson and minio/simdjson-go do not, and that is the wrong side to be on.

It is deliberately shaped like json.SyntaxError — same field, same meaning, same Error() text apart from the package name — but it cannot be an alias, because json.SyntaxError's msg field is unexported and one cannot be built from outside that package.

func (*SyntaxError) Error

func (e *SyntaxError) Error() string

type Token

type Token = json.Token

A Token is a delimiter, a string, a number, a bool, or nil — the same set encoding/json.Token holds, and the same types, so code written against one works against the other.

type UnmarshalTypeError

type UnmarshalTypeError = json.UnmarshalTypeError

UnmarshalTypeError describes a JSON value that was not appropriate for a value of a specific Go type. Its Offset field is the byte offset in the input after reading the value.

type Unmarshaler

type Unmarshaler = json.Unmarshaler

Unmarshaler is the interface implemented by types that can unmarshal a JSON description of themselves.

It is an alias for json.Unmarshaler.

type UnsupportedTypeError

type UnsupportedTypeError = json.UnsupportedTypeError

UnsupportedTypeError is returned by Marshal for a Go type that cannot be represented as JSON.

type UnsupportedValueError

type UnsupportedValueError = json.UnsupportedValueError

UnsupportedValueError is returned by Marshal for a value that cannot be represented as JSON — an infinity or a NaN.

type Value

type Value struct {
	// contains filtered or unexported fields
}

Value is one JSON value inside a document.

func GetMany

func GetMany(data []byte, paths ...[]string) []Value

GetMany returns the values at each of paths, in order.

One index, many lookups. gjson's GetMany is documented as one pass over the document for N paths, which is what it has to do because it has no index; here the document is scanned once whatever N is, and each path after that is a walk over structural positions. So the second path is nearly free and the hundredth is too.

A path that does not exist gives an Invalid Value in that position rather than an error, matching gjson. A document that does not parse gives all-Invalid.

Each path is a sequence of object keys. For anything more than that — array indices, wildcards, queries — walk with Value.Index and Value.ForEach.

func GetPath

func GetPath(data []byte, path string) Value

GetPath indexes data and returns the value at path.

It does not validate the whole document, only the part it walks through — the same contract gjson.Get has, and for the same reason: a caller pulling one field out of a payload is not asking whether the other fields are well-formed, and proving it costs four times what finding the field does. Use Parse when the answer matters.

For more than one query on the same document, index once with Parser.Scan or Parse and use Doc.Path. That is the whole point of having an index and it is where this stops being a straight loss against gjson: gjson keeps nothing, so its second query costs exactly what its first did.

one field, near the front   gjson 105 us   this  83 us
one field, near the back    gjson 105 us   this  85 us
ten fields                  gjson 634 us   this 239 us

It returns no error: a document that does not parse gives an Invalid Value, the same as a path that does not exist. Value.Exists tells them apart from a value that is there.

func (Value) All

func (v Value) All() iter.Seq2[int, Value]

All ranges over the elements of an array, or over nothing for any other kind.

The range form of Value.ForEach, for `for i, e := range v.All()`.

func (Value) Bool

func (v Value) Bool() bool

Bool returns a boolean value.

func (Value) Decode

func (v Value) Decode(out any) error

Decode stores this value in the value pointed to by v.

It is Unmarshal for a part of a document, so a large payload can be navigated to the field that matters and only that field decoded.

func (Value) Exists

func (v Value) Exists() bool

Exists reports whether the value was found.

Example

A missing key yields a Value that does not exist rather than an error, so a path can be walked without checking every step.

package main

import (
	"fmt"

	"github.com/sebishogun/simdjson"
)

func main() {
	doc, _ := simdjson.Parse([]byte(`{"a":{"b":1}}`))

	fmt.Println(doc.Get("a", "b").Exists())
	fmt.Println(doc.Get("a", "zzz").Exists())
	fmt.Println(doc.Get("nope", "deeper").Exists())
}

Output:
true
false
false

func (Value) Float

func (v Value) Float() float64

Float returns a number value as a float64.

func (Value) ForEach

func (v Value) ForEach(fn func(Value) bool)

ForEach calls fn for each element of an array until it returns false.

Example

Iterating an array without building one.

package main

import (
	"fmt"

	"github.com/sebishogun/simdjson"
)

func main() {
	doc, _ := simdjson.Parse([]byte(`{"scores":[10,20,30]}`))

	total := int64(0)
	doc.Get("scores").ForEach(func(v simdjson.Value) bool {
		total += v.Int()
		return true
	})
	fmt.Println(total)
}

Output:
60

func (Value) ForEachKey

func (v Value) ForEachKey(fn func(string, Value) bool)

ForEachKey calls fn for each field of an object until it returns false.

Example

Iterating an object's fields.

package main

import (
	"fmt"

	"github.com/sebishogun/simdjson"
)

func main() {
	doc, _ := simdjson.Parse([]byte(`{"a":1,"b":2}`))

	doc.Root().ForEachKey(func(k string, v simdjson.Value) bool {
		fmt.Printf("%s=%d\n", k, v.Int())
		return true
	})
}

Output:
a=1
b=2

func (Value) Get

func (v Value) Get(path ...string) Value

Get returns the value at a path relative to v.

The same walk as Doc.Get but starting here, which is what makes a Value worth holding on to: find the interesting subtree once, then ask it questions. gjson's Result.Get is the same idea.

A path that does not exist gives an Invalid Value; see Value.Exists.

func (Value) Index

func (v Value) Index(n int) Value

Index returns the nth element of an array.

func (Value) Int

func (v Value) Int() int64

Int returns a number value as an int64.

func (Value) IsNull

func (v Value) IsNull() bool

IsNull reports whether the value is JSON null.

func (Value) Key

func (v Value) Key(name string) Value

Key returns the value of a field in an object.

The scan walks the object's structural entries rather than its bytes, so passing over a large nested value costs one bracket match instead of a parse. A missing key gives an Invalid Value; see Value.Exists.

func (Value) Keys

func (v Value) Keys() iter.Seq[string]

Keys ranges over the field names of an object.

func (Value) Kind

func (v Value) Kind() Kind

Kind returns the value's type.

func (Value) Len

func (v Value) Len() int

Len returns the number of elements in an array or fields in an object.

func (Value) Members

func (v Value) Members() iter.Seq2[string, Value]

Members ranges over the fields of an object, or over nothing for any other kind.

The range form of Value.ForEachKey. It is not called All because an object and an array are different shapes and returning the same type for both would mean an index nobody wants or a key that does not exist.

func (Value) Path

func (v Value) Path(path string) Value

Path returns the value at a dot-separated path, relative to v.

A component that does not exist gives an Invalid Value; see Value.Exists.

func (Value) Raw

func (v Value) Raw() []byte

Raw returns the value's bytes, undecoded, pointing into the document.

func (Value) String

func (v Value) String() string

String returns a string value's contents, or "" for anything else.

A string with no escape is returned without copying the bytes out of the document; one with an escape is decoded into a new string.

func (Value) StringNoCopy

func (v Value) StringNoCopy() string

StringNoCopy is Value.String without the copy: for a string that needs no unescaping, the result points into the document rather than at bytes of its own.

The whole point of a two-stage parser is that the bytes are already there and already known to be a string, so copying them out is work nobody asked for. minio/simdjson-go exposes the same thing as WithCopyStrings(false), fastjson as StringBytes, and gjson's Result.Raw is a substring of the input by construction.

The cost is a lifetime the compiler will not check for you. The returned string aliases the slice passed to Parse, so it is only valid while that slice is unmodified and reachable, and writing through the original slice changes a string — which Go otherwise guarantees cannot happen. Use it when the document outlives the strings taken from it and both stay in one function; use Value.String anywhere the string escapes.

A string containing an escape sequence has nothing to alias, because its decoded form is not present in the document. Those are unescaped and copied exactly as Value.String does, so this is never wrong, only sometimes no faster.

func (Value) Time

func (v Value) Time() time.Time

Time parses a string value as an RFC 3339 timestamp.

The zero Time if the value is not a string or does not parse. gjson's Result.Time is the same, and it is here because timestamps in JSON are strings often enough that everybody writes this function.

func (Value) Values

func (v Value) Values() iter.Seq[Value]

Values ranges over the elements of an array or the field values of an object.