jsons

README

go-jsons

A universal JSON merge library for Go.

(test coverage: 100.0%)

Installation

go get github.com/qjebbs/go-jsons

Usage

a := []byte(`{"a":1}`)
b := []byte(`{"b":[1]}`)
c := []byte(`{"b":[2]}`)
got, err := jsons.Merge(a, b, c) // got = []byte(`{"a":1,"b":[1,2]}`)

Accepted input

• string: path to a local file
• []string: paths of local files
• []byte: content of a file
• [][]byte: content list of files
• io.Reader: content reader
• []io.Reader: content readers

Merge rules

The strandard merger is intuitive and easy to understand:

• Simple values (string, number, boolean) are overwritten by later ones.
• Container values (object, array) are merged recursively.

To work with complex contents, you may create a custom merger to applies more options:

var myMerger = jsons.NewMerger(
	jsons.WithMergeBy("tag"),
	jsons.WithMergeByAndRemove("_tag"),
	jsons.WithOrderByAndRemove("_order"),
)
myMerger.Merge("a.json", "b.json")

which means:

• Elements with same tag or _tag in an array will be merged.
• Elements in an array will be sorted by the value of _order field, the smaller ones are in front.

_tag and _order fields will be removed after merge, according to the codes above.

Suppose we have...

a.json:

{
  "log": {"level": "debug"},
  "inbounds": [{"tag": "in-1"}],
  "outbounds": [{"_order": 100, "tag": "out-1"}],
  "route": {"rules": [
    {"_tag":"rule1","inbound":["in-1"],"outbound":"out-1"}
  ]}
}

b.json:

{
  "log": {"level": "error"},
  "outbounds": [{"_order": -100, "tag": "out-2"}],
  "route": {"rules": [
    {"_tag":"rule1","inbound":["in-1.1"],"outbound":"out-1.1"}
  ]}
}

Output:

{
  // level field is overwritten by the latter value
  "log": {"level": "error"},
  "inbounds": [{"tag": "in-1"}],
  "outbounds": [
    // Although out-2 is a latecomer, but it's in 
    // the front due to the smaller "_order"
    {"tag": "out-2"},
    {"tag": "out-1"}
  ],
  "route": {"rules": [
    // 2 rules are merged into one due to the same "_tag",
    // outbound field is overwritten during the merging
    {"inbound":["in-1","in-1.1"],"outbound":"out-1.1"}
  ]}
}

Custom preprocessors

You can also register custom preprocessors to modify the content before merge, for example:

func ExampleMerger_withPreprocessor() {
	a := []byte(`{"array": ["ONE", "TWO"]}`)
	b := []byte(`{"array": ["THREE"]}`)
	m := jsons.NewMerger(jsons.WithPreprocessor(func(key string, value interface{}) interface{} {
		if str, ok := value.(string); ok {
			switch str {
			case "ONE":
				return 1
			case "TWO":
				return 2
			case "THREE":
				return 3
			}
		}
		return value
	}))
	got, err := m.MergeAs(jsons.FormatJSON, a, b)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(string(got))
	// Output: {"array":[1,2,3]}
}

Load from other formats

go-jsons allows you to extend it to load other formats easily.

For example, to load from YAML files and merge to JSON:

package main

import (
	"fmt"

	"github.com/qjebbs/go-jsons"
	// goccy/go-yaml is able to use json.Unmarshaler
	"github.com/goccy/go-yaml"
)

func ExampleMerger_RegisterLoader() {
	const FormatYAML jsons.Format = "yaml"
	m := jsons.NewMerger()
	m.RegisterOrderedLoader(
		FormatYAML,
		[]string{".yaml", ".yml"},
		func(b []byte) (*jsons.OrderedMap, error) {
			// YAML fields order will be kept
			m := jsons.NewOrderedMap()
			err := yaml.UnmarshalWithOptions(
				b, m,
				// important
				yaml.UseJSONUnmarshaler(),
			)
			if err != nil {
				return nil, err
			}
			return m, nil
		},
	)
	a := []byte(`{"a":1,"z":1}`)    // json
	b := []byte("b: 1\nc: 1\nd: 1") // yaml
	got, err := m.Merge(a, b)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(got))
	// Output: {"a":1,"z":1,"b":1,"c":1,"d":1}
}

Why not support remote files?

Here are some considerations:

• It makes the your program support remote file unexpectedly, which may be a security risk.
• Users need to choose their own strategy for loading remote files, not hard-coded logic in the library
• You can still merge downloaded content by []byte or io.Reader

Documentation

Overview

Package jsons is a universal JSON merge library for Go.

Example (Merge)

a := []byte(`{"a":1}`)
b := []byte(`{"b":[1]}`)
c := []byte(`{"b":[2]}`)
d := []byte(`{"c":null}`)

got, err := jsons.Merge(a, b, c, d)
if err != nil {
	panic(err)
}

fmt.Println(string(got))

Output:
{"a":1,"b":[1,2],"c":null}

Example (MergeAdvanced)

a := []byte(`
	{
      "log": {"level": "debug"},
      "inbounds": [{"tag": "in-1"}],
      "outbounds": [{"_order": 100, "tag": "out-1"}],
      "route": {"rules": [
        {"_tag":"rule1","inbound":["in-1"],"outbound":"out-1"}
	  ]}
	}`)
b := []byte(`
	{
      "log": {"level": "error"},
      "outbounds": [{"_order": -100, "tag": "out-2"}],
      "route": {"rules": [
        {"_tag":"rule1","inbound":["in-1.1"],"outbound":"out-1.1"}
      ]}
    }`)

var m = jsons.NewMerger(
	jsons.WithMergeBy("tag"),
	jsons.WithMergeByAndRemove("_tag"),
	jsons.WithOrderByAndRemove("_order"),
	jsons.WithIndent("", "  "),
)
got, err := m.Merge(a, b)
if err != nil {
	panic(err)
}

fmt.Println(string(got))

Output:
{
  "log": {
    "level": "error"
  },
  "inbounds": [
    {
      "tag": "in-1"
    }
  ],
  "outbounds": [
    {
      "tag": "out-2"
    },
    {
      "tag": "out-1"
    }
  ],
  "route": {
    "rules": [
      {
        "inbound": [
          "in-1",
          "in-1.1"
        ],
        "outbound": "out-1.1"
      }
    ]
  }
}

Index

• Variables
• func Merge(inputs ...interface{}) ([]byte, error)
• type Format
• type LoadFunc
• type LoadOrderedFunc
• type Merger
  • func NewMerger(options ...Option) *Merger
  • func (m *Merger) Extensions(formatNames ...Format) ([]string, error)
  • func (m *Merger) Merge(inputs ...interface{}) ([]byte, error)
  • func (m *Merger) MergeAs(format Format, inputs ...interface{}) ([]byte, error)
  • func (m *Merger) RegisterLoader(name Format, extensions []string, fn LoadFunc) error
  • func (m *Merger) RegisterOrderedLoader(name Format, extensions []string, fn LoadOrderedFunc) error
• type Option
  • func WithIndent(prefix, indent string) Option
  • func WithMergeBy(name string) Option
  • func WithMergeByAndRemove(name string) Option
  • func WithOrderBy(name string) Option
  • func WithOrderByAndRemove(name string) Option
  • func WithPreprocessor(preprocessor PreprocessorFunc) Option
  • func WithTypeOverride(override bool) Option
• type OrderedMap
• type PreprocessorFunc

Examples

• Package (Merge)
• Package (MergeAdvanced)
• Merger (WithPreprocessor)
• Merger.RegisterLoader
• Merger.RegisterOrderedLoader
• WithTypeOverride

Variables

var NewOrderedMap = ordered.New

NewOrderedMap is an alias of ordered.New

Functions

func Merge

func Merge(inputs ...interface{}) ([]byte, error)

Merge merges input JSONs into a single JSON.

The default merger does only simple merging, which means:

• Simple values (string, number, boolean) are overwritten by later ones.
• Container values (object, array) are merged recursively.

Accepted Input:

• string: path to a local file
• []string: paths of local files
• []byte: content of a file
• [][]byte: content list of files
• io.Reader: content reader
• []io.Reader: content readers

If you need complex merging, create a custom merger with options.

Types

type Format

type Format string

Format is the supported format of mergers

const (
	FormatAuto Format = "auto"
	FormatJSON Format = "json"
)

built-in formats

type LoadFunc

type LoadFunc func([]byte) (map[string]interface{}, error)

LoadFunc load the input bytes to map[string]interface{}

type LoadOrderedFunc

type LoadOrderedFunc func([]byte) (*OrderedMap, error)

LoadOrderedFunc load the input bytes to *OrderedMap, which keeps the fields order

type Merger

type Merger struct {
	// contains filtered or unexported fields
}

Merger is the json merger

Example (WithPreprocessor)

a := []byte(`{"array": ["ONE", "TWO"]}`)
b := []byte(`{"array": ["THREE"]}`)
m := jsons.NewMerger(jsons.WithPreprocessor(func(key string, value interface{}) interface{} {
	if str, ok := value.(string); ok {
		switch str {
		case "ONE":
			return 1
		case "TWO":
			return 2
		case "THREE":
			return 3
		}
	}
	return value
}))
got, err := m.MergeAs(jsons.FormatJSON, a, b)
if err != nil {
	fmt.Println(err)
	return
}
fmt.Println(string(got))

Output:
{"array":[1,2,3]}

func NewMerger

func NewMerger(options ...Option) *Merger

NewMerger returns a new Merger

func (*Merger) Extensions

func (m *Merger) Extensions(formatNames ...Format) ([]string, error)

Extensions get supported extensions of given formats. If formatNames is empty or contains FormatAuto, it returns all extensions.

func (*Merger) Merge

func (m *Merger) Merge(inputs ...interface{}) ([]byte, error)

Merge merges inputs into a single json.

It detects the format by file extension, or try all mergers if no extension found

Accepted Input:

• string: path to a local file
• []string: paths of local files
• []byte: content of a file
• [][]byte: content list of files
• io.Reader: content reader
• []io.Reader: content readers

func (*Merger) MergeAs

func (m *Merger) MergeAs(format Format, inputs ...interface{}) ([]byte, error)

MergeAs loads inputs of the specific format and merges into a single json.

Accepted Input:

• string: path to a local file
• []string: paths of local files
• []byte: content of a file
• [][]byte: content list of files
• io.Reader: content reader
• []io.Reader: content readers

func (*Merger) RegisterLoader

func (m *Merger) RegisterLoader(name Format, extensions []string, fn LoadFunc) error

RegisterLoader register a new format loader. The fields order is not guaranteed between merges due to the use of map[string]interface{}.

Example

const FormatTOML jsons.Format = "toml"
m := jsons.NewMerger()
m.RegisterLoader(
	FormatTOML,
	[]string{".toml"},
	func(b []byte) (map[string]interface{}, error) {
		m := make(map[string]interface{})
		// err := toml.Unmarshal(b, &m)
		// if err != nil {
		// 	return nil, err
		// }
		return m, nil
	},
)

func (*Merger) RegisterOrderedLoader

func (m *Merger) RegisterOrderedLoader(name Format, extensions []string, fn LoadOrderedFunc) error

RegisterOrderedLoader register a new format loader that loads data into an ordered map, who keeps the fields order between merges.

Example

const FormatYAML jsons.Format = "yaml"
m := jsons.NewMerger()
m.RegisterOrderedLoader(
	FormatYAML,
	[]string{".yaml", ".yml"},
	func(b []byte) (*jsons.OrderedMap, error) {
		m := jsons.NewOrderedMap()
		// "github.com/goccy/go-yaml" is recommended since it's able to use json.Unmarshaler
		// err := yaml.UnmarshalWithOptions(
		// 	b, m,
		// 	yaml.UseJSONUnmarshaler(), // important
		// )
		// if err != nil {
		// 	return nil, err
		// }
		return m, nil
	},
)

type Option

type Option func(m *Merger)

Option is the option for merger

func WithIndent

func WithIndent(prefix, indent string) Option

WithIndent sets the indent options for merged output.

func WithMergeBy

func WithMergeBy(name string) Option

WithMergeBy is the merge by field for slice sort rule

func WithMergeByAndRemove

func WithMergeByAndRemove(name string) Option

WithMergeByAndRemove is the merge by field for slice merge rule

func WithOrderBy

func WithOrderBy(name string) Option

WithOrderBy is the order by field for slice sort rule

func WithOrderByAndRemove

func WithOrderByAndRemove(name string) Option

WithOrderByAndRemove is the order by field for slice merge rule

func WithPreprocessor

func WithPreprocessor(preprocessor PreprocessorFunc) Option

WithPreprocessor adds a preprocessor function to preprocess values before merging.

func WithTypeOverride

func WithTypeOverride(override bool) Option

WithTypeOverride sets whether to override the type when merging.

Example

a := []byte(`{"a":1}`)
b := []byte(`{"a":false}`)

m := jsons.NewMerger(
	jsons.WithTypeOverride(true),
)
got, err := m.Merge(a, b)
if err != nil {
	panic(err)
}
fmt.Println(string(got))

Output:
{"a":false}

type OrderedMap

type OrderedMap = ordered.Map

OrderedMap is an alias of ordered.Map

type PreprocessorFunc

type PreprocessorFunc func(key string, value interface{}) interface{}

PreprocessorFunc is the type of preprocessor function, which can be used to preprocess values before merging.