json

README

go-json

Fast JSON encoder/decoder compatible with encoding/json for Go

Roadmap

* version ( expected release date )

* v0.9.0
 |
 | while maintaining compatibility with encoding/json, we will add convenient APIs
 |
 v
* v1.0.0

We are accepting requests for features that will be implemented between v0.9.0 and v.1.0.0. If you have the API you need, please submit your issue here.

Features

• Drop-in replacement of encoding/json
• Fast ( See Benchmark section )
• Flexible customization with options
• Coloring the encoded string
• Can propagate context.Context to MarshalJSON or UnmarshalJSON
• Can dynamically filter the fields of the structure type-safely

Installation

go get github.com/3JoB/go-json

How to use

Replace import statement from encoding/json to github.com/3JoB/go-json

-import "encoding/json"
+import "github.com/3JoB/go-json"

JSON library comparison

name	encoder	decoder	compatible with encoding/json
encoding/json	yes	yes	N/A
json-iterator/go	yes	yes	partial
easyjson	yes	yes	no
gojay	yes	yes	no
segmentio/encoding/json	yes	yes	partial
jettison	yes	no	no
simdjson-go	no	yes	no
goccy/go-json	yes	yes	yes

• json-iterator/go isn't compatible with encoding/json in many ways (e.g. https://github.com/json-iterator/go/issues/229 ), but it hasn't been supported for a long time.
• segmentio/encoding/json is well supported for encoders, but some are not supported for decoder APIs such as Token ( streaming decode )

Other libraries

• jingo

I tried the benchmark but it didn't work. Also, it seems to panic when it receives an unexpected value because there is no error handling...

• ffjson

Benchmarking gave very slow results. It seems that it is assumed that the user will use the buffer pool properly. Also, development seems to have already stopped

Benchmarks

$ cd benchmarks
$ go test -bench .

Encode

Decode

Fuzzing

go-json-fuzz is the repository for fuzzing tests. If you run the test in this repository and find a bug, please commit to corpus to go-json-fuzz and report the issue to go-json.

How it works

go-json is very fast in both encoding and decoding compared to other libraries. It's easier to implement by using automatic code generation for performance or by using a dedicated interface, but go-json dares to stick to compatibility with encoding/json and is the simple interface. Despite this, we are developing with the aim of being the fastest library.

Here, we explain the various speed-up techniques implemented by go-json.

Basic technique

The techniques listed here are the ones used by most of the libraries listed above.

Buffer reuse

Since the only value required for the result of json.Marshal(interface{}) ([]byte, error) is []byte, the only value that must be allocated during encoding is the return value []byte .

Also, as the number of allocations increases, the performance will be affected, so the number of allocations should be kept as low as possible when creating []byte.

Therefore, there is a technique to reduce the number of times a new buffer must be allocated by reusing the buffer used for the previous encoding by using sync.Pool.

Finally, you allocate a buffer that is as long as the resulting buffer and copy the contents into it, you only need to allocate the buffer once in theory.

type buffer struct {
    data []byte
}

var bufPool = sync.Pool{
    New: func() interface{} {
        return &buffer{data: make([]byte, 0, 1024)}
    },
}

buf := bufPool.Get().(*buffer)
data := encode(buf.data) // reuse buf.data

newBuf := make([]byte, len(data))
copy(newBuf, buf)

buf.data = data
bufPool.Put(buf)

Elimination of reflection

As you know, the reflection operation is very slow.

Therefore, using the fact that the address position where the type information is stored is fixed for each binary ( we call this typeptr ), we can use the address in the type information to call a pre-built optimized process.

For example, you can get the address to the type information from interface{} as follows and you can use that information to call a process that does not have reflection.

To process without reflection, pass a pointer (unsafe.Pointer) to the value is stored.

type emptyInterface struct {
    typ unsafe.Pointer
    ptr unsafe.Pointer
}

var typeToEncoder = map[uintptr]func(unsafe.Pointer)([]byte, error){}

func Marshal(v interface{}) ([]byte, error) {
    iface := (*emptyInterface)(unsafe.Pointer(&v)
    typeptr := uintptr(iface.typ)
    if enc, exists := typeToEncoder[typeptr]; exists {
        return enc(iface.ptr)
    }
    ...
}

※ In reality, typeToEncoder can be referenced by multiple goroutines, so exclusive control is required.

Unique speed-up technique

Encoder

Do not escape arguments of Marshal

json.Marshal and json.Unmarshal receive interface{} value and they perform type determination dynamically to process. In normal case, you need to use the reflect library to determine the type dynamically, but since reflect.Type is defined as interface, when you call the method of reflect.Type, The reflect's argument is escaped.

Therefore, the arguments for Marshal and Unmarshal are always escaped to the heap. However, go-json can use the feature of reflect.Type while avoiding escaping.

reflect.Type is defined as interface, but in reality reflect.Type is implemented only by the structure rtype defined in the reflect package. For this reason, to date reflect.Type is the same as *reflect.rtype.

Therefore, by directly handling *reflect.rtype, which is an implementation of reflect.Type, it is possible to avoid escaping because it changes from interface to using struct.

The technique for working with *reflect.rtype directly from go-json is implemented at rtype.go

Also, the same technique is cut out as a library ( https://github.com/3JoB/go-reflect )

Initially this feature was the default behavior of go-json. But after careful testing, I found that I passed a large value to json.Marshal() and if the argument could not be assigned to the stack, it could not be properly escaped to the heap (a bug in the Go compiler).

Therefore, this feature will be provided as an optional until this issue is resolved.

To use it, add NoEscape like MarshalNoEscape()

Encoding using opcode sequence

I explained that you can use typeptr to call a pre-built process from type information.

In other libraries, this dedicated process is processed by making it an function calling like anonymous function, but function calls are inherently slow processes and should be avoided as much as possible.

Therefore, go-json adopted the Instruction-based execution processing system, which is also used to implement virtual machines for programming language.

If it is the first type to encode, create the opcode ( instruction ) sequence required for encoding. From the second time onward, use typeptr to get the cached pre-built opcode sequence and encode it based on it. An example of the opcode sequence is shown below.

json.Marshal(struct{
    X int `json:"x"`
    Y string `json:"y"`
}{X: 1, Y: "hello"})

When encoding a structure like the one above, create a sequence of opcodes like this:

- opStructFieldHead ( `{` )
- opStructFieldInt ( `"x": 1,` )
- opStructFieldString ( `"y": "hello"` )
- opStructEnd ( `}` )
- opEnd

※ When processing each operation, write the letters on the right.

In addition, each opcode is managed by the following structure ( Pseudo code ).

type opType int
const (
    opStructFieldHead opType = iota
    opStructFieldInt
    opStructFieldStirng
    opStructEnd
    opEnd
)
type opcode struct {
    op opType
    key []byte
    next *opcode
}

The process of encoding using the opcode sequence is roughly implemented as follows.

func encode(code *opcode, b []byte, p unsafe.Pointer) ([]byte, error) {
    for {
        switch code.op {
        case opStructFieldHead:
            b = append(b, '{')
            code = code.next
        case opStructFieldInt:
            b = append(b, code.key...)
            b = appendInt((*int)(unsafe.Pointer(uintptr(p)+code.offset)))
            code = code.next
        case opStructFieldString:
            b = append(b, code.key...)
            b = appendString((*string)(unsafe.Pointer(uintptr(p)+code.offset)))
            code = code.next
        case opStructEnd:
            b = append(b, '}')
            code = code.next
        case opEnd:
            goto END
        }
    }
END:
    return b, nil
}

In this way, the huge switch-case is used to encode by manipulating the linked list opcodes to avoid unnecessary function calls.

Opcode sequence optimization

One of the advantages of encoding using the opcode sequence is the ease of optimization. The opcode sequence mentioned above is actually converted into the following optimized operations and used.

- opStructFieldHeadInt ( `{"x": 1,` )
- opStructEndString ( `"y": "hello"}` )
- opEnd

It has been reduced from 5 opcodes to 3 opcodes ! Reducing the number of opcodees means reducing the number of branches with switch-case. In other words, the closer the number of operations is to 1, the faster the processing can be performed.

In go-json, optimization to reduce the number of opcodes itself like the above and it speeds up by preparing opcodes with optimized paths.

Change recursive call from CALL to JMP

Recursive processing is required during encoding if the type is defined recursively as follows:

type T struct {
    X int
    U *U
}

type U struct {
    T *T
}

b, err := json.Marshal(&T{
    X: 1,
    U: &U{
        T: &T{
            X: 2,
        },
    },
})
fmt.Println(string(b)) // {"X":1,"U":{"T":{"X":2,"U":null}}}

In go-json, recursive processing is processed by the operation type of opStructFieldRecursive.

In this operation, after acquiring the opcode sequence used for recursive processing, the function is not called recursively as it is, but the necessary values ​​are saved by itself and implemented by moving to the next operation.

The technique of implementing recursive processing with the JMP operation while avoiding the CALL operation is a famous technique for implementing a high-speed virtual machine.

For more details, please refer to the article ( but Japanese only ).

Dispatch by typeptr from map to slice

When retrieving the data cached from the type information by typeptr, we usually use map. Map requires exclusive control, so use sync.Map for a naive implementation.

However, this is slow, so it's a good idea to use the atomic package for exclusive control as implemented by segmentio/encoding/json ( https://github.com/segmentio/encoding/blob/master/json/codec.go#L41-L55 ).

This implementation slows down the set instead of speeding up the get, but it works well because of the nature of the library, it encodes much more for the same type.

However, as a result of profiling, I noticed that runtime.mapaccess2 accounts for a significant percentage of the execution time. So I thought if I could change the lookup from map to slice.

There is an API named typelinks defined in the runtime package that the reflect package uses internally. This allows you to get all the type information defined in the binary at runtime.

The fact that all type information can be acquired means that by constructing slices in advance with the acquired total number of type information, it is possible to look up with the value of typeptr without worrying about out-of-range access.

However, if there is too much type information, it will use a lot of memory, so by default we will only use this optimization if the slice size fits within 2Mib .

If this approach is not available, it will fall back to the atomic based process described above.

If you want to know more, please refer to the implementation here

Decoder

Dispatch by typeptr from map to slice

Like the encoder, the decoder also uses typeptr to call the dedicated process.

Faster termination character inspection using NUL character

In order to decode, you have to traverse the input buffer character by position. At that time, if you check whether the buffer has reached the end, it will be very slow.

buf : []byte type variable. holds the string passed to the decoder cursor : int64 type variable. holds the current read position

buflen := len(buf)
for ; cursor < buflen; cursor++ { // compare cursor and buflen at all times, it is so slow.
    switch buf[cursor] {
    case ' ', '\n', '\r', '\t':
    }
}

Therefore, by adding the NUL (\000) character to the end of the read buffer as shown below, it is possible to check the termination character at the same time as other characters.

for {
    switch buf[cursor] {
    case ' ', '\n', '\r', '\t':
    case '\000':
        return nil
    }
    cursor++
}

Use Boundary Check Elimination

Due to the NUL character optimization, the Go compiler does a boundary check every time, even though buf[cursor] does not cause out-of-range access.

Therefore, go-json eliminates boundary check by fetching characters for hotspot by pointer operation. For example, the following code.

func char(ptr unsafe.Pointer, offset int64) byte {
	return *(*byte)(unsafe.Pointer(uintptr(ptr) + uintptr(offset)))
}

p := (*sliceHeader)(&unsafe.Pointer(buf)).data
for {
    switch char(p, cursor) {
    case ' ', '\n', '\r', '\t':
    case '\000':
        return nil
    }
    cursor++
}

Checking the existence of fields of struct using Bitmaps

I found by the profiling result, in the struct decode, lookup process for field was taking a long time.

For example, consider decoding a string like {"a":1,"b":2,"c":3} into the following structure:

type T struct {
    A int `json:"a"`
    B int `json:"b"`
    C int `json:"c"`
}

At this time, it was found that it takes a lot of time to acquire the decoding process corresponding to the field from the field name as shown below during the decoding process.

fieldName := decodeKey(buf, cursor) // "a" or "b" or "c"
decoder, exists := fieldToDecoderMap[fieldName] // so slow
if exists {
    decoder(buf, cursor)
} else {
    skipValue(buf, cursor)
}

To improve this process, json-iterator/go is optimized so that it can be branched by switch-case when the number of fields in the structure is 10 or less (switch-case is faster than map). However, there is a risk of hash collision because the value hashed by the FNV algorithm is used for conditional branching. Also, gojay processes this part at high speed by letting the library user yourself write switch-case.

go-json considers and implements a new approach that is different from these. I call this bitmap field optimization.

The range of values ​​per character can be represented by [256]byte. Also, if the number of fields in the structure is 8 or less, int8 type can represent the state of each field. In other words, it has the following structure.

• Base ( 8bit ): 00000000
• Key "a": 00000001 ( assign key "a" to the first bit )
• Key "b": 00000010 ( assign key "b" to the second bit )
• Key "c": 00000100 ( assign key "c" to the third bit )

Bitmap structure is the following

        | key index(0) |
------------------------
 0      | 00000000     |
 1      | 00000000     |
~~      |              |
97 (a)  | 00000001     |
98 (b)  | 00000010     |
99 (c)  | 00000100     |
~~      |              |
255     | 00000000     |

You can think of this as a Bitmap with a height of 256 and a width of the maximum string length in the field name. In other words, it can be represented by the following type .

[maxFieldKeyLength][256]int8

When decoding a field character, check whether the corresponding character exists by referring to the pre-built bitmap like the following.

var curBit int8 = math.MaxInt8 // 11111111

c := char(buf, cursor)
bit := bitmap[keyIdx][c]
curBit &= bit
if curBit == 0 {
    // not found field
}

If curBit is not 0 until the end of the field string, then the string is You may have hit one of the fields. But the possibility is that if the decoded string is shorter than the field string, you will get a false hit.

• input: {"a":1}

type T struct {
    X int `json:"abc"`
}

※ Since a is shorter than abc, it can decode to the end of the field character without curBit being 0.

Rest assured. In this case, it doesn't matter because you can tell if you hit by comparing the string length of a with the string length of abc.

Finally, calculate the position of the bit where 1 is set and get the corresponding value, and you're done.

Using this technique, field lookups are possible with only bitwise operations and access to slices.

go-json uses a similar technique for fields with 9 or more and 16 or less fields. At this time, Bitmap is constructed as [maxKeyLen][256]int16 type.

Currently, this optimization is not performed when the maximum length of the field name is long (specifically, 64 bytes or more) in addition to the limitation of the number of fields from the viewpoint of saving memory usage.

Others

I have done a lot of other optimizations. I will find time to write about them. If you have any questions about what's written here or other optimizations, please visit the #go-json channel on gophers.slack.com .

Reference

Regarding the story of go-json, there are the following articles in Japanese only.

• https://speakerdeck.com/goccy/zui-su-falsejsonraiburariwoqiu-mete
• https://engineering.mercari.com/blog/entry/1599563768-081104c850/

Looking for Sponsors

I'm looking for sponsors this library. This library is being developed as a personal project in my spare time. If you want a quick response or problem resolution when using this library in your project, please register as a sponsor. I will cooperate as much as possible. Of course, this library is developed as an MIT license, so you can use it freely for free.

License

MIT

Documentation

Index

• Variables
• func Compact(dst *bytes.Buffer, src []byte) error
• func HTMLEscape(dst *bytes.Buffer, src []byte)
• func Indent(dst *bytes.Buffer, src []byte, prefix, indent string) error
• func Marshal(v any) ([]byte, error)
• func MarshalContext(ctx context.Context, v any, optFuncs ...EncodeOptionFunc) ([]byte, error)
• func MarshalIndent(v any, prefix, indent string) ([]byte, error)
• func MarshalIndentWithOption(v any, prefix, indent string, optFuncs ...EncodeOptionFunc) ([]byte, error)
• func MarshalNoEscape(v any) ([]byte, error)
• func MarshalWithOption(v any, optFuncs ...EncodeOptionFunc) ([]byte, error)
• func Unmarshal(data []byte, v any) error
• func UnmarshalContext(ctx context.Context, data []byte, v any, optFuncs ...DecodeOptionFunc) error
• func UnmarshalNoEscape(data []byte, v any, optFuncs ...DecodeOptionFunc) error
• func UnmarshalWithOption(data []byte, v any, optFuncs ...DecodeOptionFunc) error
• func Valid(data []byte) bool
• type ColorFormat
• type ColorScheme
• type DecodeOption
• type DecodeOptionFunc
  • func DecodeFieldPriorityFirstWin() DecodeOptionFunc
• type Decoder
  • func NewDecoder(r io.Reader) *Decoder
  • func (d *Decoder) Buffered() io.Reader
  • func (d *Decoder) Decode(v any) error
  • func (d *Decoder) DecodeContext(ctx context.Context, v any) error
  • func (d *Decoder) DecodeWithOption(v any, optFuncs ...DecodeOptionFunc) error
  • func (d *Decoder) DisallowUnknownFields()
  • func (d *Decoder) InputOffset() int64
  • func (d *Decoder) More() bool
  • func (d *Decoder) Token() (Token, error)
  • func (d *Decoder) UseNumber()
• type Delim
• type EncodeOption
• type EncodeOptionFunc
  • func Colorize(scheme *ColorScheme) EncodeOptionFunc
  • func Debug() EncodeOptionFunc
  • func DebugDOT(w io.WriteCloser) EncodeOptionFunc
  • func DebugWith(w io.Writer) EncodeOptionFunc
  • func DisableHTMLEscape() EncodeOptionFunc
  • func DisableNormalizeUTF8() EncodeOptionFunc
  • func EnableCamelCase() EncodeOptionFunc
  • func UnorderedMap() EncodeOptionFunc
• type Encoder
  • func NewEncoder(w io.Writer) *Encoder
  • func (e *Encoder) Encode(v any) error
  • func (e *Encoder) EncodeContext(ctx context.Context, v any, optFuncs ...EncodeOptionFunc) error
  • func (e *Encoder) EncodeWithOption(v any, optFuncs ...EncodeOptionFunc) error
  • func (e *Encoder) SetEscapeHTML(on bool)
  • func (e *Encoder) SetIndent(prefix, indent string)
• type FieldQuery
  • func BuildFieldQuery(fields ...FieldQueryString) (*FieldQuery, error)
• type FieldQueryString
• type InvalidUTF8Error
• type InvalidUnmarshalError
• type Marshaler
• type MarshalerContext
• type MarshalerError
• type Number
• type Path
  • func CreatePath(p string) (*Path, error)
  • func (p *Path) Extract(data []byte, optFuncs ...DecodeOptionFunc) ([][]byte, error)
  • func (p *Path) Get(src, dst any) error
  • func (p *Path) PathString() string
  • func (p *Path) RootSelectorOnly() bool
  • func (p *Path) Unmarshal(data []byte, v any, optFuncs ...DecodeOptionFunc) error
  • func (p *Path) UsedDoubleQuotePathSelector() bool
  • func (p *Path) UsedSingleQuotePathSelector() bool
• type PathError
• type RawMessage
• type SubFieldQuery
  • func BuildSubFieldQuery(name string) *SubFieldQuery
  • func (q *SubFieldQuery) Fields(fields ...FieldQueryString) FieldQueryString
• type SyntaxError
• type Token
• type UnmarshalFieldError
• type UnmarshalTypeError
• type Unmarshaler
• type UnmarshalerContext
• type UnsupportedTypeError
• type UnsupportedValueError

Variables

var (
	// FieldQueryFromContext get current FieldQuery from context.Context.
	FieldQueryFromContext = encoder.FieldQueryFromContext

	// SetFieldQueryToContext set current FieldQuery to context.Context.
	SetFieldQueryToContext = encoder.SetFieldQueryToContext
)

var (
	DefaultColorScheme = &ColorScheme{
		Int:       createColorFormat(fgHiMagentaColor),
		Uint:      createColorFormat(fgHiMagentaColor),
		Float:     createColorFormat(fgHiMagentaColor),
		Bool:      createColorFormat(fgHiYellowColor),
		String:    createColorFormat(fgHiGreenColor),
		Binary:    createColorFormat(fgHiRedColor),
		ObjectKey: createColorFormat(fgHiCyanColor),
		Null:      createColorFormat(fgBlueColor),
	}
)

Functions

func Compact

func Compact(dst *bytes.Buffer, src []byte) error

Compact appends to dst the JSON-encoded src with insignificant space characters elided.

func HTMLEscape

func HTMLEscape(dst *bytes.Buffer, src []byte)

HTMLEscape appends to dst the JSON-encoded src with <, >, &, U+2028 and U+2029 characters inside string literals changed to \u003c, \u003e, \u0026, \u2028, \u2029 so that the JSON will be safe to embed inside HTML <script> tags. For historical reasons, web browsers don't honor standard HTML escaping within <script> tags, so an alternative JSON encoding must be used.

func Indent

func Indent(dst *bytes.Buffer, src []byte, prefix, indent string) error

Indent appends to dst an indented form of the JSON-encoded src. Each element in a JSON object or array begins on a new, indented line beginning with prefix followed by one or more copies of indent according to the indentation nesting. The data appended to dst does not begin with the prefix nor any indentation, to make it easier to embed inside other formatted JSON data. Although leading space characters (space, tab, carriage return, newline) at the beginning of src are dropped, trailing space characters at the end of src are preserved and copied to dst. For example, if src has no trailing spaces, neither will dst; if src ends in a trailing newline, so will dst.

func Marshal

func Marshal(v any) ([]byte, error)

Marshal returns the JSON encoding of v.

Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.

Otherwise, Marshal uses the following type-dependent default encodings:

Boolean values encode as JSON booleans.

Floating point, integer, and Number values encode as JSON numbers.

String values encode as JSON strings coerced to valid UTF-8, replacing invalid bytes with the Unicode replacement rune. The angle brackets "<" and ">" are escaped to "\u003c" and "\u003e" to keep some browsers from misinterpreting JSON output as HTML. Ampersand "&" is also escaped to "\u0026" for the same reason. This escaping can be disabled using an Encoder that had SetEscapeHTML(false) called on it.

Array and slice values encode as JSON arrays, except that []byte encodes as a base64-encoded string, and a nil slice encodes as the null JSON value.

Struct values encode as JSON objects. Each exported struct field becomes a member of the object, using the field name as the object key, unless the field is omitted for one of the reasons given below.

The encoding of each struct field can be customized by the format string stored under the "json" key in the struct field's tag. The format string gives the name of the field, possibly followed by a comma-separated list of options. The name may be empty in order to specify options without overriding the default field name.

The "omitempty" option specifies that the field should be omitted from the encoding if the field has an empty value, defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string.

As a special case, if the field tag is "-", the field is always omitted. Note that a field with name "-" can still be generated using the tag "-,".

Examples of struct field tags and their meanings:

// Field appears in JSON as key "myName".
Field int `json:"myName"`

// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`

// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`

// Field is ignored by this package.
Field int `json:"-"`

// Field appears in JSON as key "-".
Field int `json:"-,"`

The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:

Int64String int64 `json:",string"`

The key name will be used if it's a non-empty string consisting of only Unicode letters, digits, and ASCII punctuation except quotation marks, backslash, and comma.

Anonymous struct fields are usually marshaled as if their inner exported fields were fields in the outer struct, subject to the usual Go visibility rules amended as described in the next paragraph. An anonymous struct field with a name given in its JSON tag is treated as having that name, rather than being anonymous. An anonymous struct field of interface type is treated the same as having that type as its name, rather than being anonymous.

The Go visibility rules for struct fields are amended for JSON when deciding which field to marshal or unmarshal. If there are multiple fields at the same level, and that level is the least nested (and would therefore be the nesting level selected by the usual Go rules), the following extra rules apply:

1) Of those fields, if any are JSON-tagged, only tagged fields are considered, even if there are multiple untagged fields that would otherwise conflict.

2) If there is exactly one field (tagged or not according to the first rule), that is selected.

3) Otherwise there are multiple fields, and all are ignored; no error occurs.

Handling of anonymous struct fields is new in Go 1.1. Prior to Go 1.1, anonymous struct fields were ignored. To force ignoring of an anonymous struct field in both current and earlier versions, give the field a JSON tag of "-".

Map values encode as JSON objects. The map's key type must either be a string, an integer type, or implement encoding.TextMarshaler. The map keys are sorted and used as JSON object keys by applying the following rules, subject to the UTF-8 coercion described for string values above:

• string keys are used directly
• encoding.TextMarshalers are marshaled
• integer keys are converted to strings

Pointer values encode as the value pointed to. A nil pointer encodes as the null JSON value.

Interface values encode as the value contained in the interface. A nil interface value encodes as the null JSON value.

Channel, complex, and function values cannot be encoded in JSON. Attempting to encode such a value causes Marshal to return an UnsupportedTypeError.

JSON cannot represent cyclic data structures and Marshal does not handle them. Passing cyclic structures to Marshal will result in an infinite recursion.

func MarshalContext

func MarshalContext(ctx context.Context, v any, optFuncs ...EncodeOptionFunc) ([]byte, error)

MarshalContext returns the JSON encoding of v with context.Context and EncodeOption.

func MarshalIndent

func MarshalIndent(v any, prefix, indent string) ([]byte, error)

MarshalIndent is like Marshal but applies Indent to format the output. Each JSON element in the output will begin on a new line beginning with prefix followed by one or more copies of indent according to the indentation nesting.

func MarshalIndentWithOption

func MarshalIndentWithOption(v any, prefix, indent string, optFuncs ...EncodeOptionFunc) ([]byte, error)

MarshalIndentWithOption is like Marshal but applies Indent to format the output with EncodeOption.

func MarshalNoEscape

func MarshalNoEscape(v any) ([]byte, error)

MarshalNoEscape returns the JSON encoding of v and doesn't escape v.

func MarshalWithOption

func MarshalWithOption(v any, optFuncs ...EncodeOptionFunc) ([]byte, error)

MarshalWithOption returns the JSON encoding of v with EncodeOption.

func Unmarshal

func Unmarshal(data []byte, v any) error

Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v. If v is nil or not a pointer, Unmarshal returns an InvalidUnmarshalError.

Unmarshal uses the inverse of the encodings that Marshal uses, allocating maps, slices, and pointers as necessary, with the following additional rules:

To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer. If the pointer is nil, Unmarshal allocates a new value for it to point to.

To unmarshal JSON into a value implementing the Unmarshaler interface, Unmarshal calls that value's UnmarshalJSON method, including when the input is a JSON null. Otherwise, if the value implements encoding.TextUnmarshaler and the input is a JSON quoted string, Unmarshal calls that value's UnmarshalText method with the unquoted form of the string.

To unmarshal JSON into a struct, Unmarshal matches incoming object keys to the keys used by Marshal (either the struct field name or its tag), preferring an exact match but also accepting a case-insensitive match. By default, object keys which don't have a corresponding struct field are ignored (see Decoder.DisallowUnknownFields for an alternative).

To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:

bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null

To unmarshal a JSON array into a slice, Unmarshal resets the slice length to zero and then appends each element to the slice. As a special case, to unmarshal an empty JSON array into a slice, Unmarshal replaces the slice with a new empty slice.

To unmarshal a JSON array into a Go array, Unmarshal decodes JSON array elements into corresponding Go array elements. If the Go array is smaller than the JSON array, the additional JSON array elements are discarded. If the JSON array is smaller than the Go array, the additional Go array elements are set to zero values.

To unmarshal a JSON object into a map, Unmarshal first establishes a map to use. If the map is nil, Unmarshal allocates a new map. Otherwise Unmarshal reuses the existing map, keeping existing entries. Unmarshal then stores key-value pairs from the JSON object into the map. The map's key type must either be any string type, an integer, implement json.Unmarshaler, or implement encoding.TextUnmarshaler.

If a JSON value is not appropriate for a given target type, or if a JSON number overflows the target type, Unmarshal skips that field and completes the unmarshaling as best it can. If no more serious errors are encountered, Unmarshal returns an UnmarshalTypeError describing the earliest such error. In any case, it's not guaranteed that all the remaining fields following the problematic one will be unmarshaled into the target object.

The JSON null value unmarshals into an interface, map, pointer, or slice by setting that Go value to nil. Because null is often used in JSON to mean “not present,” unmarshaling a JSON null into any other Go type has no effect on the value and produces no error.

When unmarshaling quoted strings, invalid UTF-8 or invalid UTF-16 surrogate pairs are not treated as an error. Instead, they are replaced by the Unicode replacement character U+FFFD.

func UnmarshalContext

func UnmarshalContext(ctx context.Context, data []byte, v any, optFuncs ...DecodeOptionFunc) error

UnmarshalContext parses the JSON-encoded data and stores the result in the value pointed to by v. If you implement the UnmarshalerContext interface, call it with ctx as an argument.

func UnmarshalNoEscape

func UnmarshalNoEscape(data []byte, v any, optFuncs ...DecodeOptionFunc) error

func UnmarshalWithOption

func UnmarshalWithOption(data []byte, v any, optFuncs ...DecodeOptionFunc) error

func Valid

func Valid(data []byte) bool

Valid reports whether data is a valid JSON encoding.

Types

type ColorFormat

type ColorFormat = encoder.ColorFormat

type ColorScheme

type ColorScheme = encoder.ColorScheme

type DecodeOption

type DecodeOption = decoder.Option

type DecodeOptionFunc

type DecodeOptionFunc func(*DecodeOption)

func DecodeFieldPriorityFirstWin

func DecodeFieldPriorityFirstWin() DecodeOptionFunc

DecodeFieldPriorityFirstWin in the default behavior, go-json, like encoding/json, will reflect the result of the last evaluation when a field with the same name exists. This option allow you to change this behavior. this option reflects the result of the first evaluation if a field with the same name exists. This behavior has a performance advantage as it allows the subsequent strings to be skipped if all fields have been evaluated.

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

func NewDecoder

func NewDecoder(r io.Reader) *Decoder

NewDecoder returns a new decoder that reads from r.

The decoder introduces its own buffering and may read data from r beyond the JSON values requested.

func (*Decoder) Buffered

func (d *Decoder) Buffered() io.Reader

Buffered returns a reader of the data remaining in the Decoder's buffer. The reader is valid until the next call to Decode.

func (*Decoder) Decode

func (d *Decoder) Decode(v any) error

Decode reads the next JSON-encoded value from its input and stores it in the value pointed to by v.

See the documentation for Unmarshal for details about the conversion of JSON into a Go value.

func (*Decoder) DecodeContext

func (d *Decoder) DecodeContext(ctx context.Context, v any) error

DecodeContext reads the next JSON-encoded value from its input and stores it in the value pointed to by v with context.Context.

func (*Decoder) DecodeWithOption

func (d *Decoder) DecodeWithOption(v any, optFuncs ...DecodeOptionFunc) error

func (*Decoder) DisallowUnknownFields

func (d *Decoder) DisallowUnknownFields()

DisallowUnknownFields causes the Decoder to return an error when the destination is a struct and the input contains object keys which do not match any non-ignored, exported fields in the destination.

func (*Decoder) InputOffset

func (d *Decoder) InputOffset() int64

func (*Decoder) More

func (d *Decoder) More() bool

func (*Decoder) Token

func (d *Decoder) Token() (Token, error)

func (*Decoder) UseNumber

func (d *Decoder) UseNumber()

UseNumber causes the Decoder to unmarshal a number into an interface{} as a Number instead of as a float64.

type Delim

type Delim = json.Delim

A Delim is a JSON array or object delimiter, one of [ ] { or }.

type EncodeOption

type EncodeOption = encoder.Option

type EncodeOptionFunc

type EncodeOptionFunc func(*EncodeOption)

func Colorize

func Colorize(scheme *ColorScheme) EncodeOptionFunc

Colorize add an identifier for coloring to the string of the encoded result.

func Debug

func Debug() EncodeOptionFunc

Debug outputs debug information when panic occurs during encoding.

func DebugDOT

func DebugDOT(w io.WriteCloser) EncodeOptionFunc

DebugDOT sets the destination to write opcodes graph.

func DebugWith

func DebugWith(w io.Writer) EncodeOptionFunc

DebugWith sets the destination to write debug messages.

func DisableHTMLEscape

func DisableHTMLEscape() EncodeOptionFunc

DisableHTMLEscape disables escaping of HTML characters ( '&', '<', '>' ) when encoding string.

func DisableNormalizeUTF8

func DisableNormalizeUTF8() EncodeOptionFunc

DisableNormalizeUTF8 By default, when encoding string, UTF8 characters in the range of 0x80 - 0xFF are processed by applying \ufffd for invalid code and escaping for \u2028 and \u2029. This option disables this behaviour. You can expect faster speeds by applying this option, but be careful. encoding/json implements here: https://github.com/golang/go/blob/6178d25fc0b28724b1b5aec2b1b74fc06d9294c7/src/encoding/json/encode.go#L1067-L1093.

func EnableCamelCase

func EnableCamelCase() EncodeOptionFunc

EnableCamelCase convert the keys to camel case when encoding a struct.

func UnorderedMap

func UnorderedMap() EncodeOptionFunc

UnorderedMap doesn't sort when encoding map type.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

An Encoder writes JSON values to an output stream.

func NewEncoder

func NewEncoder(w io.Writer) *Encoder

NewEncoder returns a new encoder that writes to w.

func (*Encoder) Encode

func (e *Encoder) Encode(v any) error

Encode writes the JSON encoding of v to the stream, followed by a newline character.

See the documentation for Marshal for details about the conversion of Go values to JSON.

func (*Encoder) EncodeContext

func (e *Encoder) EncodeContext(ctx context.Context, v any, optFuncs ...EncodeOptionFunc) error

EncodeContext call Encode with context.Context and EncodeOption.

func (*Encoder) EncodeWithOption

func (e *Encoder) EncodeWithOption(v any, optFuncs ...EncodeOptionFunc) error

EncodeWithOption call Encode with EncodeOption.

func (*Encoder) SetEscapeHTML

func (e *Encoder) SetEscapeHTML(on bool)

SetEscapeHTML specifies whether problematic HTML characters should be escaped inside JSON quoted strings. The default behavior is to escape &, <, and > to \u0026, \u003c, and \u003e to avoid certain safety problems that can arise when embedding JSON in HTML.

In non-HTML settings where the escaping interferes with the readability of the output, SetEscapeHTML(false) disables this behavior.

func (*Encoder) SetIndent

func (e *Encoder) SetIndent(prefix, indent string)

SetIndent instructs the encoder to format each subsequent encoded value as if indented by the package-level function Indent(dst, src, prefix, indent). Calling SetIndent("", "") disables indentation.

type FieldQuery

type FieldQuery = encoder.FieldQuery

FieldQuery you can dynamically filter the fields in the structure by creating a FieldQuery, adding it to context.Context using SetFieldQueryToContext and then passing it to MarshalContext. This is a type-safe operation, so it is faster than filtering using map[string]interface{}.

func BuildFieldQuery

func BuildFieldQuery(fields ...FieldQueryString) (*FieldQuery, error)

BuildFieldQuery builds FieldQuery by fieldName or sub field query. First, specify the field name that you want to keep in structure type. If the field you want to keep is a structure type, by creating a sub field query using BuildSubFieldQuery, you can select the fields you want to keep in the structure. This description can be written recursively.

type FieldQueryString

type FieldQueryString = encoder.FieldQueryString

type InvalidUTF8Error

type InvalidUTF8Error = errors.InvalidUTF8Error

Before Go 1.2, an InvalidUTF8Error was returned by Marshal when attempting to encode a string value with invalid UTF-8 sequences. As of Go 1.2, Marshal instead coerces the string to valid UTF-8 by replacing invalid bytes with the Unicode replacement rune U+FFFD.

Deprecated: No longer used; kept for compatibility.

type InvalidUnmarshalError

type InvalidUnmarshalError = errors.InvalidUnmarshalError

An InvalidUnmarshalError describes an invalid argument passed to Unmarshal. (The argument to Unmarshal must be a non-nil pointer.)

type Marshaler

type Marshaler interface {
	MarshalJSON() ([]byte, error)
}

Marshaler is the interface implemented by types that can marshal themselves into valid JSON.

type MarshalerContext

type MarshalerContext interface {
	MarshalJSON(context.Context) ([]byte, error)
}

MarshalerContext is the interface implemented by types that can marshal themselves into valid JSON with context.Context.

type MarshalerError

type MarshalerError = errors.MarshalerError

A MarshalerError represents an error from calling a MarshalJSON or MarshalText method.

type Number

type Number = json.Number

A Number represents a JSON number literal.

type Path

type Path struct {
	// contains filtered or unexported fields
}

Path represents JSON Path.

func CreatePath

func CreatePath(p string) (*Path, error)

CreatePath creates JSON Path.

JSON Path rule $ : root object or element. The JSON Path format must start with this operator, which refers to the outermost level of the JSON-formatted string. . : child operator. You can identify child values using dot-notation. .. : recursive descent. [] : subscript operator. If the JSON object is an array, you can use brackets to specify the array index. [*] : all objects/elements for array.

Reserved words must be properly escaped when included in Path.

Escape Rule single quote style escape: e.g.) `$['a.b'].c` double quote style escape: e.g.) `$."a.b".c`

func (*Path) Extract

func (p *Path) Extract(data []byte, optFuncs ...DecodeOptionFunc) ([][]byte, error)

Extract extracts a specific JSON string.

func (*Path) Get

func (p *Path) Get(src, dst any) error

Get extract and substitute the value of the part corresponding to JSON Path from the input value.

func (*Path) PathString

func (p *Path) PathString() string

PathString returns original JSON Path string.

func (*Path) RootSelectorOnly

func (p *Path) RootSelectorOnly() bool

RootSelectorOnly whether only the root selector ($) is used.

func (*Path) Unmarshal

func (p *Path) Unmarshal(data []byte, v any, optFuncs ...DecodeOptionFunc) error

Unmarshal extract and decode the value of the part corresponding to JSON Path from the input data.

func (*Path) UsedDoubleQuotePathSelector

func (p *Path) UsedDoubleQuotePathSelector() bool

UsedSingleQuotePathSelector whether double quote-based escaping was done when building the JSON Path.

func (*Path) UsedSingleQuotePathSelector

func (p *Path) UsedSingleQuotePathSelector() bool

UsedSingleQuotePathSelector whether single quote-based escaping was done when building the JSON Path.

type PathError

type PathError = errors.PathError

type RawMessage

type RawMessage = json.RawMessage

RawMessage is a raw encoded JSON value. It implements Marshaler and Unmarshaler and can be used to delay JSON decoding or precompute a JSON encoding.

type SubFieldQuery

type SubFieldQuery struct {
	// contains filtered or unexported fields
}

func BuildSubFieldQuery

func BuildSubFieldQuery(name string) *SubFieldQuery

BuildSubFieldQuery builds sub field query.

func (*SubFieldQuery) Fields

func (q *SubFieldQuery) Fields(fields ...FieldQueryString) FieldQueryString

type SyntaxError

type SyntaxError = errors.SyntaxError

A SyntaxError is a description of a JSON syntax error.

type Token

type Token = json.Token

A Token holds a value of one of these types:

Delim, for the four JSON delimiters [ ] { }
bool, for JSON booleans
float64, for JSON numbers
Number, for JSON numbers
string, for JSON string literals
nil, for JSON null

type UnmarshalFieldError

type UnmarshalFieldError = errors.UnmarshalFieldError

An UnmarshalFieldError describes a JSON object key that led to an unexported (and therefore unwritable) struct field.

Deprecated: No longer used; kept for compatibility.

type UnmarshalTypeError

type UnmarshalTypeError = errors.UnmarshalTypeError

An UnmarshalTypeError describes a JSON value that was not appropriate for a value of a specific Go type.

type Unmarshaler

type Unmarshaler interface {
	UnmarshalJSON([]byte) error
}

Unmarshaler is the interface implemented by types that can unmarshal a JSON description of themselves. The input can be assumed to be a valid encoding of a JSON value. UnmarshalJSON must copy the JSON data if it wishes to retain the data after returning.

By convention, to approximate the behavior of Unmarshal itself, Unmarshalers implement UnmarshalJSON([]byte("null")) as a no-op.

type UnmarshalerContext

type UnmarshalerContext interface {
	UnmarshalJSON(context.Context, []byte) error
}

UnmarshalerContext is the interface implemented by types that can unmarshal with context.Context a JSON description of themselves.

type UnsupportedTypeError

type UnsupportedTypeError = errors.UnsupportedTypeError

An UnsupportedTypeError is returned by Marshal when attempting to encode an unsupported value type.

type UnsupportedValueError

type UnsupportedValueError = errors.UnsupportedValueError