Documentation
¶
Overview ¶
Package jsonsplit provides JSON functionality that can dynamically switch the underlying implementation between jsonv1 and jsonv2. The purpose of this package is to provide a gradual means for migrating from v1 to v2 and detecting which options (if any) need to be specified in order to maintain backwards compatibility.
Whether it is safe to directly use v2 is dependent on both static properties of the Go types being serialized and also dynamic properties of the input JSON text that is being unmarshaled. Many of the changed behaviors in v2 can be directly specified on Go struct fields to ensure that the struct type is represented in the same way in both v1 and v2. Alternatively, some options may need to be specified when calling jsonv2.Marshal or jsonv2.Unmarshal to preserve a particular v1 behavior.
Example usage and migration ¶
1. Replace existing calls of jsonv1.Unmarshal with jsonsplit.Unmarshal. By default, jsonsplit calls jsonv1, so this is identical behavior.
2. Configure jsonsplit to call both v1 and v2 and report any differences:
func init() { // Specify that when a difference is detected, // to auto-detect which options are causing the difference. jsonsplit.GlobalCodec.AutoDetectOptions = true // Log every time we detect a difference between v1 and v2. jsonsplit.GlobalCodec.ReportDifference(func(d jsonsplit.Difference) { slog.Warn("detected jsonv1-to-jsonv2 difference", "diff", d) }) // Specify that we try both v1 and v2 with some probability, // but to always return v1 results. jsonsplit.GlobalCodec.SetMarshalCallRatio( jsonsplit.OnlyCallV1, // 90% of the time jsonsplit.CallBothButReturnV1, // 10% of the time 0.1, ) // Publish an expvar under the "jsonsplit" name. jsonsplit.Publish() }
While we can detect differences in behavior between v1 and v2, the semantic behavior is still identical to v1 since both call modes are configured to return the v1 result.
3. Run the program and monitor logs and metrics. Let's suppose that through logging, we discover for this Go type:
type User struct { FirstName string `json:"firstName"` LastName string `json:"lastName"` }
that unmarshal is being provided a JSON input like:
{ "FIRSTNAME": "John", "LASTNAME": "Doe" }
which happens to work fine in v1 because jsonv1 uses case-insensitive matching by default, while jsonv2 use case-sensitive matching by default.
There are two ways to resolve this difference.
4a. (Option 1) We can mark every Go struct field as being case insensitive:
type User struct { FirstName string `json:"firstName,case:ignore"` LastName string `json:"lastName,case:ignore"` }
This has the advantage of making sure this struct operates the same regardless of whether it is called by jsonv1 or jsonv2. However, this has the disadvantage of requiring tedious modification of every field in the Go struct and may not even be possible if the declaration of the Go type is not within your control.
4b. (Option 2) We can call jsonv2 with jsonv2.MatchCaseInsensitiveNames:
... := jsonsplit.Marshal(v, jsonv2.MatchCaseInsensitiveNames(true))
This has the advantage of being able alter the behavior of unmarshal at the call site, affecting all types that it recursively reaches. This has the disadvantage that the type will behave differently depending on whether it is called by jsonv1 and jsonv2 with the default behavior.
5. Let the program run for a while and gradually increase the ratio of trying both v1 and v2. If we detect no more differences, then we have decent confidence that we have handled all the relevant differences in behavior between v1 and v2. We can now gradually switch to using v2 exclusively:
func init() { // Specify that we start to return v2 results with some probability. jsonsplit.GlobalCodec.SetMarshalCallRatio( jsonsplit.CallBothButReturnV1, // 90% of the time jsonsplit.OnlyCallV2, // 10% of the time 0.1, ) }
This will occasionally return the results of v2 and you can verify that your program continues to function as expected.
6. After increasing exclusive use of v2 to 100% and still not encountering any issues, we can now confidently replace jsonsplit.Unmarshal with jsonv2.Unmarshal (and possibly with jsonv2.MatchCaseInsensitiveNames if we need to maintain backwards compatibility or drop it if we decide to allow a breaking change).
Example ¶
Example of calling marshal and unmarshal with v1 semantics, but being able to detect differences between v1 and v2.
package main import ( "fmt" "log" "slices" "github.com/go-json-experiment/jsonsplit" ) func main() { c := jsonsplit.Codec{ // When differences between v1 and v2 are detected, try to also // detect which specific options are causing the difference. AutoDetectOptions: true, // Print out the detected differences between v1 and v2. ReportDifference: func(d jsonsplit.Difference) { switch d.Func { case "Marshal": fmt.Printf("Marshal difference detected:\n"+ "\tGoValue: %+v\n"+ "\tJSONValueV1: %s\n"+ "\tJSONValueV2: %s\n"+ "\tOptions: %v\n", d.GoValue, d.JSONValueV1, d.JSONValueV2, slices.Collect(d.OptionNames())) case "Unmarshal": fmt.Printf("Unmarshal difference detected:\n"+ "\tJSONValue: %s\n"+ "\tGoValueV1: %+v\n"+ "\tGoValueV2: %+v\n"+ "\tOptions: %v\n", d.JSONValue, d.GoValueV1, d.GoValueV2, slices.Collect(d.OptionNames())) } }, } // Specify that marshal/unmarshal should call both v1 and v2, // but continue to return the results of v1. c.SetMarshalCallMode(jsonsplit.CallBothButReturnV1) c.SetUnmarshalCallMode(jsonsplit.CallBothButReturnV1) const in = `{"FIRSTNAME":"John","LASTNAME":"Doe","lastName":"Dupe"}` type User struct { FirstName string `json:"firstName"` LastName string `json:"lastName"` Age int `json:"age,omitempty"` Aliases []string `json:"tags"` } var u User // Unmarshal according to v1 semantics, which will: // - match JSON object names case-insensitively // - allow duplicate JSON object names if err := c.Unmarshal([]byte(in), &u); err != nil { log.Fatal(err) } // Marshal according to v1 semantics, which will: // - emit Age since omitempty works with integers in v1 // - emit Aliases as a JSON null instead of a [] if _, err := c.Marshal(u); err != nil { log.Fatal(err) } }
Output: Unmarshal difference detected: JSONValue: {"FIRSTNAME":"John","LASTNAME":"Doe","lastName":"Dupe"} GoValueV1: &{FirstName:John LastName:Dupe Age:0 Aliases:[]} GoValueV2: &{FirstName: LastName:Dupe Age:0 Aliases:[]} Options: [jsontext.AllowDuplicateNames jsonv2.MatchCaseInsensitiveNames] Marshal difference detected: GoValue: {FirstName:John LastName:Dupe Age:0 Aliases:[]} JSONValueV1: {"firstName":"John","lastName":"Dupe","tags":null} JSONValueV2: {"firstName":"John","lastName":"Dupe","age":0,"tags":[]} Options: [jsonv1.OmitEmptyWithLegacySemantics jsonv2.FormatNilSliceAsNull]
Index ¶
- Variables
- func Marshal(v any, o ...jsonv2.Options) (b []byte, err error)
- func Publish()
- func Unmarshal(b []byte, v any, o ...jsonv2.Options) error
- type CallMode
- type Codec
- func (c *Codec) Helper()
- func (c *Codec) Marshal(v any, o ...jsonv2.Options) (b []byte, err error)
- func (c *Codec) MarshalCallRatio() (mode1, mode2 CallMode, ratio float64)
- func (c *Codec) SetMarshalCallMode(mode CallMode)
- func (c *Codec) SetMarshalCallRatio(mode1, mode2 CallMode, ratio float64)
- func (c *Codec) SetUnmarshalCallMode(mode CallMode)
- func (c *Codec) SetUnmarshalCallRatio(mode1, mode2 CallMode, ratio float64)
- func (c *Codec) Unmarshal(b []byte, v any, o ...jsonv2.Options) (err error)
- func (c *Codec) UnmarshalCallRatio() (mode1, mode2 CallMode, ratio float64)
- type CodecMetrics
- type Difference
- type SizeHistogram
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNotCloneable = errors.New("Go value could not be cloned")
ErrNotCloneable reports that Codec.Unmarshal was unable to clone the output Go value, so it could not unmarshal with both v1 and v2 in order to properly check for any differences.
[Codec.ReportDifference] is still called and this sentinel error is specified as [Difference.ErrorV1] or [Difference.ErrorV2]. If [Difference.ErrorV1] is this error, then [Difference.GoValueV2] is the input value prior to unmarshal and [Difference.GoValueV1] is nil. If [Difference.ErrorV2] is this error, then [Difference.GoValueV1] is the input value prior to unmarshal and [Difference.GoValueV2] is nil.
Functions ¶
func Marshal ¶
Marshal marshals from v with either jsonv1.Marshal or jsonv2.Marshal depending on the mode specified in Codec.SetMarshalCallRatio on the GlobalCodec variable.
func Publish ¶
func Publish()
Publish calls expvar.Publish with CodecMetrics.ExpVar under the name "jsonsplit".
func Unmarshal ¶
Unmarshal unmarshals into v with either jsonv1.Unmarshal or jsonv2.Unmarshal depending on the mode specified in Codec.SetUnmarshalCallRatio on the GlobalCodec variable.
Types ¶
type CallMode ¶
type CallMode int
CallMode configures how Codec.Marshal and Codec.Unmarshal delegates calls to either v1 or v2 functionality.
const ( // OnlyCallV1 specifies to only call v1 functionality. OnlyCallV1 CallMode = iota // CallV1ButUponErrorReturnV2 specifies to call v1 by default, // but only when an error occurs, to call v2 and return its result instead. CallV1ButUponErrorReturnV2 // CallBothButReturnV1 specifies to call both v1 and v2 functionality, // but to return the results for v1. CallBothButReturnV1 // CallBothButReturnV2 specifies to call both v1 and v2 functionality, // but to return the results for v2. CallBothButReturnV2 // CallV2ButUponErrorReturnV1 specifies to call v2 by default, // but only when an error occurs, to call v1 and return its result instead. CallV2ButUponErrorReturnV1 // OnlyCallV2 specifies to only call v2 functionality. OnlyCallV2 )
type Codec ¶
type Codec struct { // AutoDetectOptions specifies whether to automatically detect which // [jsontext], [jsonv1], or [jsonv2] options are needed to preserve // identical behavior between v1 and v2 once a difference has been detected. // // Auto-detection is relatively slow and will need to run marshal/unmarshal // many extra times. In performance sensitive systems, // configure [Codec.SetMarshalCallRatio] and [Codec.SetUnmarshalCallRatio] // such that [CallBothButReturnV1] or [CallBothButReturnV2] call modes // occur with relatively low probability. AutoDetectOptions bool // ReportDifference is a custom function to report detected differences // in marshal or unmarshal. If nil, structured differences are ignored. // The fields in [Difference] alias the call arguments for marshal/unmarshal // and should therefore avoid leaking beyond the function call. // Must be set before any [Codec.Marshal] or [Codec.Unmarshal] calls. ReportDifference func(Difference) // EqualJSONValues is a custom function to compare JSON values after marshal. // If nil, it uses [bytes.Equal]. EqualJSONValues func(jsontext.Value, jsontext.Value) bool // EqualGoValues is a custom function to compare Go values after unmarshal. // If nil, it uses [reflect.DeepEqual]. EqualGoValues func(any, any) bool // EqualErrors is a custom function to compare errors from marshal or unmarshal. // If nil, it only checks whether the errors are both non-nil or both nil. EqualErrors func(error, error) bool // CloneGoValue is a custom function to deeply clone an arbitrary Go value // for use as the output for calling unmarshal. // If nil (or the function returns nil), then it clones any // pointers to a zero'd value by simply allocating a new one. CloneGoValue func(v any) any CodecMetrics // contains filtered or unexported fields }
Codec configures how to execute marshal and unmarshal calls. The exported fields must be set before concurrent use. The zero value is ready for use and by default will OnlyCallV1.
func (*Codec) Helper ¶
func (c *Codec) Helper()
Helper marks the calling function as a helper function. When producing a Difference, that function will be skipped when deriving the caller for marshal or unmarshal.
func (*Codec) Marshal ¶
Marshal marshals from v with either jsonv1.Marshal or jsonv2.Marshal depending on the mode specified in Codec.SetMarshalCallRatio. If both v1 and v2 are called, it checks whether any differences are detected in the serialized JSON output values.
The specified options o is applied on top of the default v1 or v2 options. If o is exactly equal to jsonv1.DefaultOptionsV1, then this calls jsonv1std.Marshal instead of jsonv1.Marshal when operating in v1 mode. This allows for detection of differences between jsonv1std and jsonv1.
func (*Codec) MarshalCallRatio ¶
MarshalCallRatio retrieves the mode and ratio parameters previously set by Codec.SetMarshalCallRatio.
func (*Codec) SetMarshalCallMode ¶
SetMarshalCallMode specifies the CallMode for marshaling. By default, marshal will use OnlyCallV1. This is safe to call concurrently with Codec.Marshal.
func (*Codec) SetMarshalCallRatio ¶
SetMarshalCallRatio sets the ratio of Codec.Marshal calls that will use the marshal functionality of v1, v2, or both.
The ratio must be within 0 and 1, where:
- 0.0 means to use mode1 100% of the time and mode2 0% of the time.
- 0.1 means to use mode1 90% of the time and mode2 10% of the time.
- 0.5 means to use mode1 50% of the time and mode2 50% of the time.
- 0.9 means to use mode1 10% of the time and mode2 90% of the time.
- 1.0 means to use mode1 0% of the time and mode2 100% of the time.
For example:
// This configures marshal to call v1 90% of the time, // but call both both v1 and v2 10% of the time // (while still returning the result of v1). codec.SetMarshalCallRatio(OnlyCallV1, CallBothButReturnV1, 0.1)
By default, marshal will use OnlyCallV1. This is safe to call concurrently with Codec.Marshal.
func (*Codec) SetUnmarshalCallMode ¶
SetUnmarshalCallMode specifies the CallMode for unmarshaling. By default, unmarshal will only use OnlyCallV1. This is safe to call concurrently with Codec.Unmarshal.
func (*Codec) SetUnmarshalCallRatio ¶
SetUnmarshalCallRatio sets the ratio of Codec.Unmarshal calls that will use the unmarshal functionality of v1, v2, or both.
The ratio must be within 0 and 1, where:
- 0.0 means to use mode1 100% of the time and mode2 0% of the time.
- 0.1 means to use mode1 90% of the time and mode2 10% of the time.
- 0.5 means to use mode1 50% of the time and mode2 50% of the time.
- 0.9 means to use mode1 10% of the time and mode2 90% of the time.
- 1.0 means to use mode1 0% of the time and mode2 100% of the time.
For example:
// This configures unmarshal to call v1 90% of the time, // but call both both v1 and v2 10% of the time // (while still returning the result of v1). codec.SetUnmarshalCallRatio(OnlyCallV1, CallBothButReturnV1, 0.1)
By default, unmarshal will only use OnlyCallV1. This is safe to call concurrently with Codec.Unmarshal.
func (*Codec) Unmarshal ¶
Unmarshal unmarshals to v with either jsonv1.Unmarshal or jsonv2.Unmarshal depending on the mode specified in Codec.SetUnmarshalCallRatio. If both v1 and v2 are called, it checks whether any differences are detected in the deserialized Go output values.
The specified options o is applied on top of the default v1 or v2 options. If o is exactly equal to jsonv1.DefaultOptionsV1, then this calls jsonv1std.Unmarshal instead of jsonv1.Unmarshal when operating in v1 mode. This allows for detection of differences between jsonv1std and jsonv1.
func (*Codec) UnmarshalCallRatio ¶
UnmarshalCallRatio retrieves the mode and ratio parameters previously set by Codec.SetUnmarshalCallRatio.
type CodecMetrics ¶
type CodecMetrics struct { // NumMarshalTotal is the total number of [Codec.Marshal] calls. NumMarshalTotal expvar.Int // NumMarshalErrors is the total number of [Codec.Marshal] calls // that returned an error. NumMarshalErrors expvar.Int // NumMarshalOnlyCallV1 is the number of [Codec.Marshal] calls // that only delegated the call to [jsonv1.Marshal]. NumMarshalOnlyCallV1 expvar.Int // NumMarshalOnlyCallV2 is the number of [Codec.Marshal] calls // that only delegated the call to [jsonv2.Marshal]. NumMarshalOnlyCallV2 expvar.Int // NumMarshalCallBoth is the number of [Codec.Marshal] calls // that called both [jsonv1.Marshal] and [jsonv2.Marshal]. NumMarshalCallBoth expvar.Int // NumMarshalReturnV1 is the number of [Codec.Marshal] calls // that used the result of [jsonv1.Marshal]. NumMarshalReturnV1 expvar.Int // NumMarshalReturnV2 is the number of [Codec.Marshal] calls // that used the result of [jsonv2.Marshal]. NumMarshalReturnV2 expvar.Int // NumMarshalDiffs is the number of times that [Codec.Marshal] detected // a difference between the outputs of [jsonv1.Marshal] and [jsonv2.Marshal]. NumMarshalDiffs expvar.Int // ExecTimeMarshalV1Nanos is the total number of nanoseconds // spent in a [jsonv1.Marshal] call when comparing both v1 and v2. // It excludes time spent only calling v1. ExecTimeMarshalV1Nanos expvar.Int // ExecTimeMarshalV2Nanos is the total number of nanoseconds // spent in a [jsonv2.Marshal] call when comparing both v1 and v2. // It excludes time spent only calling v2. ExecTimeMarshalV2Nanos expvar.Int // MarshalSizeHistogram is a histogram of JSON input sizes from [Codec.Marshal] // regardless of whether a difference is detected. MarshalSizeHistogram SizeHistogram // MarshalCallerHistogram is a histogram of callers to [Codec.Marshal] // whenever a difference is detected. MarshalCallerHistogram expvar.Map // MarshalOptionHistogram is a histogram of JSON options // that could be specified to [Codec.Marshal] to avoid a difference. MarshalOptionHistogram expvar.Map // NumUnmarshalTotal is the total number of [Codec.Unmarshal] calls. NumUnmarshalTotal expvar.Int // NumUnmarshalErrors is the total number of [Codec.Unmarshal] calls // that returned an error. NumUnmarshalErrors expvar.Int // NumUnmarshalMerge is the total number of [Codec.Unmarshal] calls // where the output argument is a pointer to a non-zero value. NumUnmarshalMerge expvar.Int // NumUnmarshalOnlyCallV1 is the number of [Codec.Unmarshal] calls // that only delegated the call to [jsonv1.Unmarshal]. NumUnmarshalOnlyCallV1 expvar.Int // NumUnmarshalOnlyCallV2 is the number of [Codec.Unmarshal] calls // that only delegated the call to [jsonv2.Unmarshal]. NumUnmarshalOnlyCallV2 expvar.Int // NumUnmarshalCallBoth is the number of [Codec.Unmarshal] calls // that called both [jsonv1.Unmarshal] and [jsonv2.Unmarshal]. NumUnmarshalCallBoth expvar.Int // NumUnmarshalCallBothSkipped is the number of [Codec.Unmarshal] calls // that could not call both v1 and v2 because of some problem. NumUnmarshalCallBothSkipped expvar.Int // NumUnmarshalReturnV1 is the number of [Codec.Unmarshal] calls // that used the result of [jsonv1.Unmarshal]. NumUnmarshalReturnV1 expvar.Int // NumUnmarshalReturnV2 is the number of [Codec.Unmarshal] calls // that used the result of [jsonv2.Unmarshal]. NumUnmarshalReturnV2 expvar.Int // NumUnmarshalDiffs is the number of times that [Codec.Unmarshal] detected // a difference between the outputs of [jsonv1.Unmarshal] and [jsonv2.Unmarshal]. // // This includes counts in [CodecMetrics.NumUnmarshalCallBothSkipped] // as inability to check for differences is treated as a difference // to avoid false assurance that there are no differences. NumUnmarshalDiffs expvar.Int // ExecTimeUnmarshalV1Nanos is the total number of nanoseconds // spent in a [jsonv1.Unmarshal] call when comparing both v1 and v2. ExecTimeUnmarshalV1Nanos expvar.Int // ExecTimeUnmarshalV2Nanos is the total number of nanoseconds // spent in a [jsonv2.Unmarshal] call when comparing both v1 and v2. ExecTimeUnmarshalV2Nanos expvar.Int // UnmarshalSizeHistogram is a histogram of JSON input sizes to [Codec.Unmarshal] // regardless of whether a difference is detected. UnmarshalSizeHistogram SizeHistogram // UnmarshalCallerHistogram is a histogram of callers to [Codec.Unmarshal] // whenever a difference is detected. UnmarshalCallerHistogram expvar.Map // UnmarshalOptionHistogram is a histogram of JSON options // that could be specified to [Codec.Unmarshal] to avoid a difference. UnmarshalOptionHistogram expvar.Map }
CodecMetrics contains metrics about marshal and unmarshal calls.
func (*CodecMetrics) ExpVar ¶
func (c *CodecMetrics) ExpVar() expvar.Var
ExpVar returns an expvar mapping of all metrics. It reports variables with the snake case form of each field in CodecMetrics.
type Difference ¶
type Difference struct { // Caller is the function name and relative line offset of the caller. // For example, "path/to/package.Function+123". Caller string `json:",omitzero"` // Func is the operation and is either "Marshal" or "Unmarshal". Func string `json:",omitzero"` // GoType is the Go type being operated upon. GoType reflect.Type `json:",omitzero"` // JSONValue is the input JSON value provided to an unmarshal call. JSONValue jsontext.Value `json:",omitzero"` // JSONValueV1 is the output JSON value produced by a v1 marshal call. JSONValueV1 jsontext.Value `json:",omitzero"` // JSONValueV2 is the output JSON value produced by a v2 marshal call. JSONValueV2 jsontext.Value `json:",omitzero"` // GoValue is the input Go value provided to a marshal call. GoValue any `json:"-"` // GoValueV1 is the output Go value populated by a v1 unmarshal call. GoValueV1 any `json:"-"` // GoValueV2 is the output Go value populated by a v2 unmarshal call. GoValueV2 any `json:"-"` // ErrorV1 is the error produced by a v1 marshal/unmarshal call. ErrorV1 error `json:",omitzero"` // ErrorV2 is the error produced by a v2 marshal/unmarshal call. ErrorV2 error `json:",omitzero"` // Options is the set of options that need to be enabled // in order to resolve any behavior difference between v1 and v2. // It is only populated if [Codec.AutoDetectOptions] is enabled. Options jsonv2.Options `json:",omitzero"` }
Difference is a structured representation of the difference detected between the outputs of a v1 and v2 marshal or unmarshal call.
func (Difference) MarshalJSON ¶
func (d Difference) MarshalJSON() ([]byte, error)
MarshalJSON marshals d as JSON in a non-reversible manner and is primarily intended for logging purposes.
In particular, it uses:
- reflect.Type.String to encode a Go type
- [error.Error] to encode a Go error
- Difference.OptionNames to encode a jsonv2.Options
func (Difference) OptionNames ¶
func (d Difference) OptionNames() iter.Seq[string]
OptionNames returns an iterator over the names of all the enabled options in [Difference.Options] that resolve any behavior difference between v1 and v2.
func (Difference) String ¶
func (d Difference) String() string
String returns the difference as JSON.
type SizeHistogram ¶
SizeHistogram is a log₂ histogram of sizes. Each index i maps to a count of sizes seen within [ 2ⁱ⁻¹ : 2ⁱ ).
func (*SizeHistogram) MarshalJSON ¶
func (h *SizeHistogram) MarshalJSON() ([]byte, error)
MarshalJSON marshals the histogram as a JSON object where each name represents a size range in the format "<N{prefix}B", and each value is the count of sizes observed in that range.
The name format is as follows:
- N is the upper bound of the size range (2ⁱ) where i is modulo 10.
- {prefix} is one of "", "Ki", "Mi", "Gi", "Ti", "Pi", or "Ei", representing binary prefixes for sizes scaled by powers of 2¹⁰.
- B denotes bytes.
For example, the name "<64KiB" indicates sizes in the range [32KiB, 64KiB). Only ranges with non-zero counts are included in the JSON output.
func (*SizeHistogram) String ¶
func (h *SizeHistogram) String() string
String returns the histogram as JSON. It implements both fmt.Stringer and expvar.Var.