unstable

package
v0.0.72 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 17, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package unstable provides the JSON scanning primitives the lightning code generator's output (and the pkg/json toolkit) call into.

It is NOT a stable API. The package is exported only because generated *_unmarshal.go files — which live in other modules — must import it; its exported functions, signatures, and error sentinels may change or be removed in any release. Do not import it directly. The name says so on purpose.

Generated decoders call the exported Read*/Skip*/Decode*/ExpectNull functions and the Err* sentinels; the unexported helpers are internal to this package. The scanner is index based and avoids allocation on the common paths (unescaped strings, integers, object keys).

Index

Constants

View Source 
const MaxDepth = 10000

MaxDepth bounds how deeply the recursive walkers — DecodeValue and the validator — will descend into nested objects and arrays before giving up with ErrMaxDepth. Those walkers recurse once per nesting level, so without a bound a document of a few hundred thousand brackets exhausts the goroutine stack, and a Go stack overflow is a *fatal* error that recover cannot catch: hostile input would take the process down rather than return an error.

The limit matches encoding/json's, so input this package accepts is input encoding/json would also accept depth-wise.

Variables

View Source 
var (
	ErrInvalidJSON  = errors.New("json: invalid JSON")
	ErrTruncated    = errors.New("json: truncated JSON")
	ErrBadEscape    = errors.New("json: invalid string escape")
	ErrBadUnicode   = errors.New("json: invalid unicode escape")
	ErrBadNumber    = errors.New("json: invalid number")
	ErrExpectColon  = errors.New("json: expected ':'")
	ErrExpectObject = errors.New("json: expected '{'")
	ErrExpectArray  = errors.New("json: expected '['")
	ErrBadTime      = errors.New("json: invalid time")
	ErrKeyNotFound  = errors.New("json: key path not found")
	ErrMaxDepth     = errors.New("json: exceeded max depth")
)

Errors returned by the scanner and by generated decoders.

Functions

func CountArrayElements 

func CountArrayElements(data []byte, i int) int

CountArrayElements returns the number of top-level elements in the JSON array beginning at data[i] (data[i] must be '['), so a destination slice can be allocated once instead of grown by repeated append. It returns 0 for an empty array or when the count cannot be determined cheaply (a truncated or malformed array); the caller then simply falls back to append-driven growth, so an imperfect count is only ever a missed optimization, never a correctness problem.

Each element is skipped whole with SkipValue rather than walked byte by byte: SkipValue uses the SIMD indexStructural scanner for nested arrays/objects and indexCloseOrEscape for strings, so a structurally dense element — a nested coordinate array of many numbers, say — is jumped over in vectorized strides instead of one byte at a time. This is what makes presizing slices of arrays, objects, or strings cheap.

func CountArrayObjects 

func CountArrayObjects(data []byte, i int) int

CountArrayObjects counts the elements of a JSON array of "bracket-free" objects beginning at data[i] (data[i] must be '['). A bracket-free object has only number/bool fields — its JSON ({"a":1,"b":2}) holds no string, '[' or ']', and no nested '{' — so the array's closing ']' is the first ']' in the input and the element count is exactly the number of '{' before it, both found with vectorized byte scans. Like CountArrayScalars it is far cheaper than CountArrayElements (no per-element SkipValue) and valid only for the element shape the generator vouches for; it sizes a slice of flat numeric records — citm_catalog's price entries — without re-scanning every struct. As a presize hint a miscount on unexpected input only over- or under-allocates, never misdecodes. Returns 0 for an empty array.

func CountArrayScalars 

func CountArrayScalars(data []byte, i int) int

CountArrayScalars counts the elements of a JSON array of scalar values (numbers, booleans, or null) beginning at data[i] (data[i] must be '['). Such elements never contain a quote, comma, or bracket, so the closing ']' is the first ']' in the input and the element count is one more than the number of commas before it — both found with vectorized byte scans. It is therefore much cheaper than CountArrayElements but valid only when the element type is known (by the generator) to be a scalar. It returns 0 for an empty array.

func DecodeByteSlice added in v0.0.63 

func DecodeByteSlice(out *[]byte, data []byte, i int) (int, error)

DecodeByteSlice decodes the JSON value at data[i] into *out with encoding/json's []byte semantics, which accept two shapes: a string value is base64 (StdEncoding, decoded with the same base64.Decode call the stdlib uses, so accepted inputs and error identities match), and an array is the numeric element form (each element parsed like any uint kind, overflow wrap included) shared with every other uint slice. null yields a nil slice. Like the other slice readers it replaces *out's contents, reusing its backing when the decoded bytes fit.

On a decode error *out holds the prefix that did decode, the partial-progress semantics the readers here share — and here it is not merely a convention: reusing the backing means a failed base64 decode has already overwritten the caller's previous bytes, so leaving *out alone would report a stale length over rewritten data. encoding/json, which always decodes into a fresh buffer, leaves its target untouched instead; a caller that must keep the old value across a failed decode has to keep its own copy.

func DecodeByteSliceArena added in v0.0.63 

func DecodeByteSliceArena(out *[]byte, data []byte, i int, a *Arena) (int, error)

DecodeByteSliceArena is DecodeByteSlice with the numeric-array form's backing carved from a (see DecodeFloat64SliceArena); the //lightning:arena decoders call it. The base64 form keeps a plain make: its length comes from the string body, not a presize count, and Decode fills every byte, so there is no zeroing to save.

func DecodeFloat64Array 

func DecodeFloat64Array(out []float64, data []byte, i int) (int, error)

DecodeFloat64Array decodes a JSON array of numbers at data[i] into the fixed-size array whose backing the caller passes as out (out = arr[:]). It mirrors the generated fixed-array decoder: the array is zeroed, up to len(out) elements are decoded, extras are skipped, a short JSON array leaves the tail zero, and a JSON null leaves the array untouched. This is the per-point call for coordinate rings ([][2]float64, [][3]float64), where the generated form paid an extra call frame per coordinate.

func DecodeFloat64Slice 

func DecodeFloat64Slice(out *[]float64, data []byte, i int) (int, error)

DecodeFloat64Slice decodes a JSON array of numbers at data[i] into *out, returning the index just past the closing ']'. A JSON null sets *out to nil; a null element decodes as 0. When *out is nil the slice is presized from a vectorized comma count.

func DecodeFloat64SliceArena added in v0.0.60 

func DecodeFloat64SliceArena(out *[]float64, data []byte, i int, a *Arena) (int, error)

DecodeFloat64SliceArena is DecodeFloat64Slice with the fresh backing carved from a instead of allocated per slice (see Arena); the //lightning:arena decoders call it. Semantics are identical, including reuse: a non-nil *out keeps its existing backing and the arena is untouched.

func DecodeIntArray 

func DecodeIntArray[T intKind](out []T, data []byte, i int) (int, error)

DecodeIntArray is DecodeFloat64Array for the integer kinds; the element parse mirrors ReadInt64OrNull (inlined, as in DecodeIntSlice).

func DecodeIntSlice 

func DecodeIntSlice[T intKind](out *[]T, data []byte, i int) (int, error)

DecodeIntSlice decodes a JSON array of integers at data[i] into *out. The element parse mirrors ReadInt64OrNull byte for byte — SWAR 4-digit folds, a scalar tail, tolerated (truncated) fraction/exponent, overflow wrap — inlined into the loop so an element costs no call. The parsed int64 is converted to T exactly as the generated per-element code converted (wrapping, not saturating). A JSON null sets *out to nil; a null element decodes as 0.

func DecodeIntSliceArena added in v0.0.60 

func DecodeIntSliceArena[T intKind](out *[]T, data []byte, i int, a *Arena) (int, error)

DecodeIntSliceArena is DecodeIntSlice with the fresh backing carved from a (see DecodeFloat64SliceArena); the //lightning:arena decoders call it.

func DecodeUintArray 

func DecodeUintArray[T uintKind](out []T, data []byte, i int) (int, error)

DecodeUintArray is DecodeIntArray for the unsigned kinds; the element parse mirrors ReadUint64OrNull (inlined, as in DecodeUintSlice).

func DecodeUintSlice 

func DecodeUintSlice[T uintKind](out *[]T, data []byte, i int) (int, error)

DecodeUintSlice is DecodeIntSlice for the unsigned kinds; the element parse mirrors ReadUint64OrNull.

func DecodeUintSliceArena added in v0.0.60 

func DecodeUintSliceArena[T uintKind](out *[]T, data []byte, i int, a *Arena) (int, error)

DecodeUintSliceArena is DecodeUintSlice with the fresh backing carved from a (see DecodeFloat64SliceArena); the //lightning:arena decoders call it.

func DecodeValue 

func DecodeValue(data []byte, i int) (any, int, error)

DecodeValue decodes an arbitrary JSON value at data[i] into the standard Go representation (nil, bool, float64, string, []any, map[string]any).

It reads through this package's readers, so it inherits their accept set rather than the JSON grammar's: numbers take scanFloat's superset (a leading '+', leading zeros, an empty integer part or fraction — DecodeValue("+5") is float64(5), and so is the element of "[+5]"; see ParseFloat for the full list), and strings come back with their bytes verbatim, invalid UTF-8 included, where encoding/json coerces to U+FFFD (see ReadStringOrNull). Both leniencies are shared with Valid, which is what lets Valid promise it accepts exactly what this does.

func DecodeValueCompact 

func DecodeValueCompact(data []byte, i int) (any, int, error)

DecodeValueCompact is DecodeValue for compact JSON — input with no whitespace between tokens — skipping the inter-token whitespace scans DecodeValue makes while walking objects and arrays. The generator routes the dynamic any/map value path here for a //lightning:compact decoder. On compact input it behaves identically to DecodeValue but faster; given inter-token whitespace it may report an error.

func ExpectNull 

func ExpectNull(data []byte, i int) (int, error)

ExpectNull consumes the literal null at data[i]. The constant-string compare compiles to a single word load and compare (no allocation, no memequal call for constants <= 16 bytes) instead of four byte compares; a partial literal ("nul" at end of input) fails the bounds test and returns i, exactly as the byte-at-a-time form did.

func GrowSlice added in v0.0.58 

func GrowSlice[T any](s []T) []T

GrowSlice returns s with its capacity at least doubled, its length and contents preserved, for a decode loop that is about to append past cap(s).

It exists to bypass runtime.nextslicecap's damping: bare append doubles only while cap is under 256 elements and then grows by cap += (cap+768)>>2, about 1.25x. Since the bytes a growing slice allocates in total come to final_cap * f/(f-1), the 1.25x regime allocates roughly 5x the final size and memmoves about 4x it, where a flat 2x allocates 2x and memmoves 1x. Arrays that stay under 256 elements are unaffected either way, so this only changes the large-array regime.

func GrowSliceEst added in v0.0.61 

func GrowSliceEst[T any](s []T, start, i, end int) []T

GrowSliceEst is GrowSlice with a progress-based capacity estimate, for a decode loop that is about to append past cap(s) and knows where in the document it is. The caller has decoded len(s) elements while the scan advanced from start (the index of the array's '[') to i, in a document ending at end (len(data)). Assuming the bytes not yet consumed hold elements at the same density as the bytes already consumed, the final element count is about 

len(s) * (end - start) / (i - start)

This is a fine hint for an array that spans the rest of the document (a root array of large records), and it costs no extra scanning — all three indexes are already live in the decode loop, which is what distinguishes it from the rejected counting presizes.

The raw ratio is then padded by 1/8 (+1) to make it genuinely upper-ish. A hint one element SHORT is the worst outcome: the loop grows once more for the tail and the 2x floor below doubles a nearly-final capacity — measured on github_events (30 large records), the unpadded estimate landed at 29, the last element forced 29 -> 58, and the decode allocated MORE than flat doubling (234 KB vs 165 KB B/op). Elements are never uniform, so a near-exact estimate is common; the 12.5% pad turns it into a small over-allocation (bounded, unlike the miss it prevents, which costs a full extra backing + memmove). With the pad the same decode allocates 103 KB — under both the unpadded estimate and the flat-2x baseline.

The estimate is only ever a capacity hint, so it is clamped to [2*cap(s), 8*cap(s)]:

  • never below 2*cap(s): each grow is at least the flat doubling GrowSlice does, so no caller regresses against that behavior (an array whose remaining bytes are NOT elements — the document continues after it — under-estimates, and the floor restores plain doubling);
  • never above 8*cap(s): a nested slice early in a large document sees an estimate inflated by all the trailing non-element bytes; the ceiling bounds the over-allocation to two extra doublings per grow (8x = 2x^3), and the next grow re-estimates from better progress.

GrowSlice's floor of 4 applies to the lower bound, so for cap(s) < 2 the clamp window is [4, 16] rather than [2*cap, 8*cap] — harmless, since the generated decoders give a fresh slice a ~256-byte first-append hint and only ever grow from there.

Overflow and degenerate inputs: the product len(s)*(end-start) is computed in uint64 because both factors can be large at once (a huge element count late in a huge document overflows int64; in uint64 it is exact for any document under ~2^32 bytes even in the worst one-byte-element case, since len(s) is bounded by the bytes consumed — and were it ever to wrap, the result is still only a capacity hint inside the clamp window: a mis-sized slice, never a misdecode). Both factors are non-negative under the guards — i <= start (no progress; includes a caller passing a bad start) or end <= start (start past the document) would divide by zero or yield nonsense, so those fall back to plain doubling, as does len(s) == 0. The clamped result is at most 4*lo, which fits uint64 exactly (lo is an int); a slice so large that 8*cap(s) exceeds int is already beyond what make/append could have built, matching GrowSlice's own 2*cap(s) exposure.

func IndexCloseOrEscape 

func IndexCloseOrEscape(b []byte) int

IndexCloseOrEscape returns the index of the first '"' or '\\' byte in b, or len(b) if neither is present. It is exported (and inlinable) so generated decoders can write the object-key / string read inline at the call site — the no-escape fast path — instead of paying a ReadKey call; ReadKey stays the escape/error fallback.

func IndexCloseOrEscapeAt added in v0.0.72 

func IndexCloseOrEscapeAt(b []byte, i int) int

IndexCloseOrEscapeAt is IndexCloseOrEscape starting the scan at i and returning an ABSOLUTE index into b (len(b) if there is no '"' or '\\' at or after i). It is what generated decoders and the readers here call, because expressing the start as IndexCloseOrEscape(b[i:]) makes the caller pay for a reslice — seven instructions on arm64: the len and cap subtractions and the negative-length clamp on the base pointer — once per object key and once per string value, where handing the offset to the scanner costs one argument word and nothing in the scan itself. Measured -20.7% on a key-read-shaped micro, and the absolute result is what every caller wanted anyway.

An i past the end of b is not an error: the answer is len(b), the same as a scan that found nothing. Every implementation agrees on that, and the amd64 one needs an explicit guard to (its byte count would otherwise go negative and walk off the buffer), which TestIndexCloseOrEscapeAtPastEnd pins.

func IndexEscape 

func IndexEscape(b []byte) int

IndexEscape returns the index of the first byte that JSON string encoding must escape — a control byte < 0x20, '"' or '\\' — or len(b) if none. It is the scan behind EscapeString/EscapeStringInto: a clean run is copied out in bulk and only the escape byte at the returned index is expanded. SIMD on amd64 (SSE2/AVX2) and arm64 (NEON, or SVE2 where the core has it); SWAR elsewhere.

func IndexEscapeNonASCII added in v0.0.68 

func IndexEscapeNonASCII(b []byte) int

IndexEscapeNonASCII returns the index of the first byte that is either one JSON string encoding must escape (a control byte < 0x20, '"' or '\\') or a non-ASCII byte (>= 0x80), or len(b) if none. It is the scan behind EscapeStringInto's UTF-8 handling: the walk runs on it until the first non-ASCII byte, where one utf8.Valid call decides between the plain escape path and U+FFFD substitution — so the widened predicate costs the pure-ASCII common case only an extra OR per SIMD block over IndexEscape.

func ParseFloat 

func ParseFloat(b []byte) (float64, error)

ParseFloat parses the number in b as a float64. It takes the same Clinger fast path as the scanner — an exact mantissa with a small decimal exponent is converted with a single multiply or divide — and falls back to strconv.ParseFloat for everything else. b must be exactly one number with no surrounding whitespace; trailing bytes or an empty input yield ErrBadNumber.

What it accepts is scanFloat's grammar, which is deliberately a superset of RFC 8259's number: a leading '+' as well as '-' (ParseFloat("+5") is 5, nil), leading zeros ("01"), an empty integer part (".5") and an empty fraction ("1."). That is not laxness for its own sake — it is the accept set of every number reader in this package, so ParseFloat, Valid and a generated decoder agree on which documents are numbers; TestValidDivergesFromStdlib pins the same list from Valid's side. In the other direction it is narrower than the JSON grammar in one place: a magnitude no float64 can represent (1e309) is ErrBadNumber, since there is no value to return.

func ReadBoolOrNull 

func ReadBoolOrNull(data []byte, i int) (bool, int, error)

ReadBoolOrNull reads a JSON boolean (or null) at data[i]. The literals are matched with constant-string compares (a word load + compare each, see ExpectNull) rather than byte at a time; a partial literal returns i, as before.

func ReadFloat64OrNull 

func ReadFloat64OrNull(data []byte, i int) (float64, int, error)

ReadFloat64OrNull reads a JSON number (or null) at data[i] as a float64.

func ReadInt64OrNull 

func ReadInt64OrNull(data []byte, i int) (int64, int, error)

ReadInt64OrNull reads a JSON integer (or null) at data[i]. Fractional and exponent parts are tolerated and truncated toward zero.

func ReadKey 

func ReadKey(data []byte, i int) (string, int, error)

ReadKey reads a JSON object key (a quoted string) at data[i] without allocating. Keys are assumed not to contain backslash escapes; if they do, the slow path is taken.

func ReadNumberNoCopyOrNull 

func ReadNumberNoCopyOrNull(data []byte, i int) (string, int, error)

ReadNumberNoCopyOrNull is ReadNumberOrNull but returns a string that aliases data instead of copying it, so the caller must keep data unchanged while the result is in use.

func ReadNumberOrNull 

func ReadNumberOrNull(data []byte, i int) (string, int, error)

ReadNumberOrNull reads a JSON number (or null) at data[i] and returns its raw literal as a string — the bytes a json.Number holds — copied out verbatim, with no value produced; a JSON null yields the empty string.

The literal is validated before it is captured. Without that check the token bounds came from a scanner that only measures a run of number bytes, so a malformed literal ("1.2.3", "-", "--1", "1e") was stored intact and failed much later and far from the decode, at the first .Float64()/.Int64() — and, worse, this package's own Valid rejected documents the decoder had accepted.

The acceptance rule is agreement with ReadFloat64OrNull, not with encoding/json, because Valid checks numbers by running them through that reader (see pkg/json/valid.go): matching it is what makes "Valid accepts exactly what these decoders accept" true for json.Number fields as well. Two consequences are deliberate and pinned by TestReadNumberAcceptSetMatchesFloat64: "01" and ".5" stay accepted though encoding/json rejects them (a pre-existing, separate divergence — narrowing it here would only move the disagreement), and a literal whose magnitude overflows float64 ("1e309") is rejected though its digits are well-formed.

func ReadStringDestructiveOrNull 

func ReadStringDestructiveOrNull(data []byte, i int) (string, int, error)

ReadStringDestructiveOrNull is ReadStringNoCopyOrNull but, for a string that contains escapes, unescapes it *in place* — overwriting the escaped bytes of data with the decoded ones — instead of allocating a scratch buffer, and aliases the result. The unescaped form is never longer than the escaped body, so it fits within the body's bytes; the rest of the body is left as overwritten garbage and the closing quote (which the write cursor never reaches) still bounds the value. This DESTROYS the input document: the bytes of every escaped string are clobbered and any other alias into the same region (an overlapping nocopy value) is invalidated. It is the //lightning:destructive counterpart of the nocopy reader, for callers that own the buffer and discard it after decoding. Escape-free strings alias the input unchanged, exactly like the nocopy reader.

func ReadStringNoCopyOrNull 

func ReadStringNoCopyOrNull(data []byte, i int) (string, int, error)

ReadStringNoCopyOrNull is like ReadStringOrNull but, for strings without escapes, returns a string that aliases data rather than copying it, so the caller must keep data unchanged while the string is in use. Strings containing escapes still allocate, since they cannot be represented as a slice of the input.

func ReadStringOrNull 

func ReadStringOrNull(data []byte, i int) (string, int, error)

ReadStringOrNull reads a JSON string (or null) at data[i], copying the bytes into a fresh string.

String bytes are returned verbatim: this decodes escapes but never inspects the UTF-8 of the literal runs between them, so raw invalid UTF-8 in the input reaches the Go string unchanged ("a\xffb" stays "a\xffb"), where encoding/json coerces the same input to "a\uFFFDb". Passing the bytes through is the cheaper and the more faithful of the two — no re-encode, and a caller that cares can run utf8.ValidString — but it does mean a decoded string is not guaranteed to be well-formed UTF-8. Every string path here shares the property because they share these scanners — ReadStringNoCopyOrNull, ReadStringDestructiveOrNull, ReadKey, the dynamic DecodeValue, and decodeEscaped's literal runs — and TestStringsPassInvalidUTF8Through pins it across all of them. Only the \uXXXX decoder normalizes: an unpaired surrogate escape becomes U+FFFD, matching encoding/json, since there is no other way to encode it.

func ReadTimeLaxOrNull 

func ReadTimeLaxOrNull(data []byte, i int) (time.Time, int, error)

ReadTimeLaxOrNull reads a time.Time at data[i], accepting more shapes than ReadTimeOrNull: an RFC 3339 string with either a 'T' or a space separator and optional fractional seconds, or a Unix timestamp (in seconds, milliseconds, or microseconds) given as a JSON number or a numeric string. The result is normalized to UTC. Anything it cannot interpret returns ErrBadTime, which the "lax" decode path turns into a skipped value and an unset field.

func ReadTimeOrNull 

func ReadTimeOrNull(data []byte, i int) (time.Time, int, error)

ReadTimeOrNull reads an RFC 3339 JSON string (or null) at data[i] as a time.Time. Its authority for the grammar is time.Parse(time.RFC3339, ...): the fast path in date.go is only ever allowed to be more conservative, and everything it declines is handed to time.Parse itself, which TestReadTimeMatchesStdlibAcceptance locks over a generated date corpus. That is also what encoding/json's time.Time reduces to today — its extra RFC 3339 strictness is compiled out pending go.dev/issue/54580 — so on an escape-free timestamp the two accept the same set and produce the same instant.

The parity stops at the JSON string layer, and only in the lenient direction: this reads the string *value* (escapes decoded) and parses that, where time.Time.UnmarshalJSON parses the raw bytes between the quotes without unescaping them (a known stdlib quirk, go.dev/issue/47353). So a timestamp written with any \uXXXX escape — legal JSON denoting a legal instant, such as "2021-01-01T00:00:00\u005A" — decodes here and is rejected by the stdlib. TestReadTimeAcceptsEscapedTimestamps pins both halves of that. ReadTimeLaxOrNull inherits the same leniency by construction.

The intermediate string aliases data — safe because time.Parse retains it in neither its result nor its error (the stdlib copies into ParseError; locked by TestReadTimeErrorRetainsNoAlias) — so this allocates only the time.Time.

func ReadUint64OrNull 

func ReadUint64OrNull(data []byte, i int) (uint64, int, error)

ReadUint64OrNull reads a JSON unsigned integer (or null) at data[i].

func SameBuffer added in v0.0.66 

func SameBuffer(a, b []byte) bool

SameBuffer reports whether a and b are backed by the same array, which is how the pkg/json rewriters recognize an in-place call (out passed as in[:0]) and take the extra care that requires. It answers "same backing array", not "overlapping": a slice into the middle of another reports false, so a caller must treat false as "not proven separate" and stay on the safe path.

func SkipString 

func SkipString(data []byte, i int) (int, error)

SkipString advances past the JSON string starting at data[i] (data[i] must be '"') and returns the index just past its closing quote. Escapes are honored so an escaped quote (\") does not end the string; the scan itself does not validate or decode the escapes. It returns ErrTruncated if the closing quote is missing before the end of data.

func SkipValue 

func SkipValue(data []byte, i int) (int, error)

SkipValue advances past any JSON value starting at data[i].

Objects and dense arrays (whose first element is itself an object, array, or string) are skipped with the SIMD in-string-mask balance scan (skipContainerFast) when it is available (AVX2 on amd64): it absorbs string keys/values into one bulk pass instead of a SkipString call per string, which is a large win on the containers Get/GetPaths and unknown-field skipping walk over. A scalar-element array ('[1,2,...]') keeps the indexStructural skip, where a single vectorized scan already reaches the closing bracket and the mask path would only add per-block work. The array probe is a heuristic; a wrong guess only costs speed, never correctness — both paths are bracket balancers that return the same end index for every well-formed value.

Off the well-formed set the two paths are not interchangeable, so which one runs — and therefore what SkipValue answers on malformed input — depends on the host CPU. skipfast.go's header enumerates the divergence classes (three as of this writing) and TestSkipPathsDivergeOnMalformed pins them; that list is the authority and is deliberately not restated here, since a second copy is what let this comment go stale while claiming to be exhaustive. Every caller treats such input as an error or a presize miss.

func SkipValueStrict added in v0.0.66 

func SkipValueStrict(data []byte, i int) (int, error)

SkipValueStrict parses the single JSON value at data[i] and returns the offset just past it, or an error if the value is not well-formed JSON as this library defines it. Nothing is decoded and nothing is allocated: it is the validating counterpart of SkipValue, which is a bracket balancer and therefore accepts balanced nonsense ("[1,]", "[1 2 3]", "[1,,2]") that this rejects.

It has two callers, and they want it for opposite halves of the same property. pkg/json.Valid is exactly this walk plus a trailing-content check. And a generated decoder's ",lax" field uses it as the skip that runs after a failed decode: lax must swallow a *type* mismatch (that is the whole point of the option) while still failing on a *syntax* error, and the difference between those two is precisely "SkipValue accepted it" versus "this accepted it".

The acceptance set is this library's own, not encoding/json's — it is the set DecodeAny reads, which is what makes Valid a usable gate in front of these decoders. See pkg/json.Valid's documentation for the enumerated differences (numbers are checked by arithmetic, so 01/+1/.5/5. are accepted and 1e309 is rejected; whitespace is any byte <= 0x20; a raw control byte inside a string is accepted). Everything else is checked strictly: a trailing comma, a non-string key, a missing colon, an unknown escape, a \u without four hex digits, an unterminated string, a mismatched bracket, a bad keyword.

It is a flat loop rather than recursion: the open containers live in a bitset (one bit per level, set for an object and clear for an array) instead of on the goroutine stack, so however deeply nested the input is, checking it costs bits and not stack frames — a Go stack overflow being fatal and beyond recover's reach. MaxDepth bounds the bitset to a fixed ~1.2 KiB local that never escapes, and nesting past it returns ErrMaxDepth rather than descending.

The scalar cases delegate to the decoder's own readers wherever a reader exists that does not allocate, so the two agree by construction rather than by parallel reimplementation: numbers go through ReadFloat64OrNull (the exact tier chain decodeValue uses, overflow behavior included). Strings are the exception — the decoder's reader unescapes, and so allocates — and are checked by strictString.

The three labels are the parser's states: scanValue wants a value, scanKey wants a member's "key": prefix, and scanAfter has just finished a value and looks for a comma or the enclosing close bracket. All locals are declared up front because Go forbids a goto that jumps over a declaration.

func SkipWS 

func SkipWS(data []byte, i int) int

SkipWS advances past JSON whitespace at data[i]. The four JSON whitespace bytes (space, tab, newline, carriage return) are all <= ' ' (0x20), so a single compare classifies a byte with no memory load — measurably faster than a lookup table on every shape from a compact exit to a deep indent run. This is the hottest classification in the scanner, running before and after every value.

The compare also treats the other control bytes (0x00..0x1f) as whitespace. Those are never valid JSON between tokens, so on well-formed input the result is identical to matching the four bytes exactly; on malformed input SkipWS skips such a byte rather than stopping on it, leaving the value parser to reject the next real token. SkipWS is not called inside strings, so control bytes within string contents are unaffected.

func SkipWSCompact 

func SkipWSCompact(data []byte, i int, compact bool) int

SkipWSCompact is the compact-aware inter-token whitespace skip shared by the dynamic DecodeValue path and the pkg/json toolkit's compact variants (GetCompact, SetMany, ObjectEachCompact, …). In compact mode the input is asserted to carry no whitespace between tokens (the form compact JSON serializers emit), so it returns i unchanged; otherwise it is SkipWS. This mirrors the generator's //lightning:compact decoders, which elide exactly these inter-token skips.

func SkipWSRun 

func SkipWSRun(data []byte, i int) int

SkipWSRun advances past a whitespace run from data[i] (the caller has already established that data[i-1] and data[i-2] were whitespace, i.e. this is a run of at least two). It is the out-of-line continuation the generated decoders call only for genuine indentation runs in pretty-printed input — short skips (zero or one whitespace byte, the compact and single-space-after-token cases) are handled inline at the call site so they never pay a call. Eight bytes are classified per word (see the derivation at nws below); a full-whitespace word is skipped whole and the first word with a structural byte locates it with one trailing-zeros count.

func SwarNeedsEscape added in v0.0.63 

func SwarNeedsEscape(v uint64) uint64

SwarNeedsEscape reports (nonzero, high bit per matching lane) which of the eight packed bytes in v JSON string encoding must escape — a control byte < 0x20, '"' or '\\'. It is the one shared spelling of that predicate: indexEscapeScalar's clean-run scan and pkg/json's escape-walk per-run probe (escapeValidInto, and via SwarNeedsEscapeOrNonASCII below, EscapeStringInto) all build on it, so the escape byte set lives in one place. Pure bit math with no calls, so it inlines into all of them (the escape walks' gates depend on that — re-check -gcflags=-m if this grows).

func SwarNeedsEscapeOrNonASCII added in v0.0.68 

func SwarNeedsEscapeOrNonASCII(v uint64) uint64

SwarNeedsEscapeOrNonASCII is SwarNeedsEscape widened by the non-ASCII lanes (high bit set): a nonzero result means some lane JSON string encoding must escape OR is >= 0x80, and the LOWEST set bit marks the first such lane. It is the predicate behind EscapeStringInto's UTF-8 handling: the escape walk runs on it until the first non-ASCII byte decides (once, via utf8.Valid) whether the rest of the input is clean UTF-8 or needs U+FFFD substitution.

Contract: only the LOWEST set bit is meaningful — higher bits may be false positives. The has-less trick's `&^ v` term exists to keep a borrow out of a lane whose own value is >= 0x20 from flagging it; dropping it and OR-ing v instead makes the widened predicate one op CHEAPER than SwarNeedsEscape itself ((v-0x20·lo)|v vs (v-0x20·lo)&^v, then &hi either way). The dropped term readmits exactly two lane kinds: high-bit lanes (wanted) and a lane whose 0x20-subtraction borrowed from a LOWER underflowing lane — and such a lower lane is < 0x20, i.e. a true match below the false positive, so TrailingZeros64 never lands on the false one. Both callers (EscapeStringInto's per-run probe, indexEscapeNonASCIIScalar's word loop) take only the first match. Same inlining constraint as SwarNeedsEscape: pure bit math, no calls.

func UnescapeString 

func UnescapeString(in []byte) (string, error)

UnescapeString decodes the body of a JSON string (the bytes that sit between the surrounding quotes, with escape sequences such as \n, \" and \uXXXX still present) into the Go string it represents. When in contains no escapes the returned string aliases in directly without copying, so the caller must keep in unchanged while the result is in use; when escapes are present a new string is allocated. It shares its slow path with the JSON scanner.

func UnescapeStringInto 

func UnescapeStringInto(in, out []byte) (string, error)

UnescapeStringInto decodes the body of a JSON string like UnescapeString, but writes the decoded bytes into out instead of allocating its own buffer:

  • if in contains no escapes, the returned string aliases in and out is untouched;
  • otherwise the decode is written into out and the returned string aliases out. When cap(out) >= len(in) this allocates nothing (unescaping never lengthens a string, so the result always fits); a shorter out is grown, which allocates.

Pass out == in (e.g. in[:0]) to decode truly in place: that is safe because the write cursor never overtakes the read cursor (each escape consumes at least as many bytes as it produces) and append uses an overlap-safe memmove. Either way the returned string aliases the buffer it was built in, which the caller must keep unchanged while the result is in use; when out aliases in, in's original (escaped) bytes are overwritten.

func UnsafeStr 

func UnsafeStr(b []byte) string

UnsafeStr returns a string that aliases b without copying, so the caller must keep b unchanged while the result is in use. Exported for the same inlined read path; escaped or copied results still go through ReadKey/ReadString*.

func Unwrap 

func Unwrap(data []byte, i int) ([]byte, int, error)

Unwrap reads the JSON string at data[i] and returns the JSON document embedded in it — the backing of the "unwrap" field option. The string body is unescaped; if it is not itself JSON (its first non-whitespace byte is not the start of a JSON value) it is base64-decoded first, standard alphabet, with or without padding. A JSON null yields a nil document and no error, and the second return value is the offset just past the consumed string. The returned slice is freshly allocated and not retained by Unwrap, so values decoded out of it may safely alias it (the "nocopy" option).

Types

type Arena added in v0.0.60 

type Arena struct {
	// contains filtered or unexported fields
}

Arena is a chunked bump allocator for the backings of small scalar slices, used by the Decode*SliceArena readers that //lightning:arena decoders call. On documents shaped like a skeletal animation or a mesh — many thousands of 3–4 element []float64 fields — the per-slice make() is the decode's dominant allocation source (marine_ik: 95% of allocated objects, ~20% of CPU in mallocgc); carving those backings out of shared chunks turns tens of thousands of tiny allocations into a few hundred chunk allocations. The total bytes allocated and zeroed are unchanged (a fresh chunk is zeroed just as each make() was), so the win is precisely the removed per-object mallocgc work, plus the GC tracking that many fewer objects.

The zero value is ready to use. An Arena is not safe for concurrent use; the generated decoders declare one per UnmarshalJSON call, which also bounds its lifetime — the Arena itself is garbage as soon as the decode returns, and each chunk lives exactly as long as some result slice still references it.

That last property is the deliberate trade-off of //lightning:arena: a single surviving 3-element slice keeps its whole arenaChunkBytes chunk reachable. Callers that decode, process, and discard together (the common shape for these documents) lose nothing; callers that retain a few small slices out of a large decode should not opt in.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.