ojg

package module
v1.21.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 11, 2024 License: MIT Imports: 6 Imported by: 3

README

Build Status Coverage Status Go Report Card

Optimized JSON for Go is a high performance parser with a variety of additional JSON tools. OjG is optimized to processing huge data sets where data does not necessarily conform to a fixed structure.

Features

  • Fast JSON parser. Check out the cmd/benchmarks app in this repo.
  • Full JSONPath implemenation that operates on simple types as well as structs.
  • Generic types. Not the proposed golang generics but type safe JSON elements.
  • Fast JSON validator (7 times faster with io.Reader).
  • Fast JSON writer with a sort option (4 times faster).
  • JSON builder from JSON sources using a simple assembly plan.
  • Simple data builders using a push and pop approach.
  • Object encoding and decoding using an approach similar to that used with Oj for Ruby.
  • Simple Encoding Notation, a lazy way to write JSON omitting commas and quotes.

Using

A basic Parse:

    obj, err := oj.ParseString(`{
        "a":[
            {"x":1,"y":2,"z":3},
            {"x":2,"y":4,"z":6}
        ]
    }`)

Using JSONPath expressions:

    x, err := jp.ParseString("a[?(@.x > 1)].y")
    ys := x.Get(obj)
    // returns [4]

The oj command (cmd/oj) uses JSON path for filtering and extracting JSON elements. It also includes sorting, reformatting, and colorizing options.

$ oj -m "(@.name == 'Pete')" myfile.json

More complete examples are available in the go docs for most functions. The example for Unmarshalling interfaces demonstrates a feature that allows interfaces to be marshalled and unmarshalled.

Installation

go get github.com/ohler55/ojg
go get github.com/ohler55/ojg/cmd/oj

or just import in your .go files.

import (
    "github.com/ohler55/ojg/alt"
    "github.com/ohler55/ojg/asm"
    "github.com/ohler55/ojg/gen"
    "github.com/ohler55/ojg/jp"
    "github.com/ohler55/ojg/oj"
    "github.com/ohler55/ojg/sen"
)

To build and install the oj application:

go install ./...

The oj application can be installed with brew.

brew tap ohler55/tap
brew install oj

Benchmarks

Higher numbers (longer bars) are better.

Parse string/[]byte
       json.Unmarshal           55916 ns/op    17776 B/op    334 allocs/op
         oj.Parse               39570 ns/op    18488 B/op    429 allocs/op
   oj-reuse.Parse               17881 ns/op     5691 B/op    364 allocs/op

   oj-reuse.Parse        █████████████████████▉ 3.13
         oj.Parse        █████████▉ 1.41
       json.Unmarshal    ▓▓▓▓▓▓▓ 1.00

Parse io.Reader
       json.Decode              63029 ns/op    32449 B/op    344 allocs/op
         oj.ParseReader         34289 ns/op    22583 B/op    430 allocs/op
   oj-reuse.ParseReader         25094 ns/op     9788 B/op    365 allocs/op
         oj.TokenizeLoad        13610 ns/op     6072 B/op    157 allocs/op

         oj.TokenizeLoad ████████████████████████████████▍ 4.63
   oj-reuse.ParseReader  █████████████████▌ 2.51
         oj.ParseReader  ████████████▊ 1.84
       json.Decode       ▓▓▓▓▓▓▓ 1.00

to JSON with indentation
       json.Marshal             78762 ns/op    26978 B/op    352 allocs/op
         oj.JSON                 7662 ns/op        0 B/op      0 allocs/op
        sen.Bytes                9053 ns/op        0 B/op      0 allocs/op

         oj.JSON         ███████████████████████████████████████████████████████████████████████▉ 10.28
        sen.Bytes        ████████████████████████████████████████████████████████████▉ 8.70
       json.Marshal      ▓▓▓▓▓▓▓ 1.00

See all benchmarks

Compare Go JSON parsers

Releases

See CHANGELOG.md

Contributing
  • Provide a Pull Request off the develop branch.
  • Report a bug
  • Suggest an idea

Documentation

Overview

Package ojg is a collection of JSON tools including a validators, parsers, a full JSONPath implementation, data conversion utilities, and a simple type assembler. Most of the tools are designed for simple types although used in complex ways. Simple types in this context are data objects composed of these types. 

bool
int64
float64
string
time.Time
[]any
map[string]any

Oj

Package oj contains functions and types for parsing JSON as well as support for building simple types. Included in the oj package are: 

Parser for parsing JSON strings and streams into simple types.

Validator for validating JSON strings and streams.

Builder for building simple types.

Writer for writing data as JSON.

Gen

Package gen provides type safe generic types. They are type safe in that array and objects can only be constructed of other types in the package. The basic types are: 

Bool
Int
Float
String
Time

The collection types are Array and Object. All the types implement the Node interface which is a relatively simple interface defined primarily to restrict what can be in the collection types. The Node interface should not be used to define new generic types.

Also included in the package are a builder and parser that behave like the parser and builder in the oj package except for gen types.

Jp

Package jp provides JSONPath implementation that operations on simple go types, generic (gen package), and public struct with public members. Get, set, and delete operations can be evaluated on data. When needed reflection is used to follow a path.

Alt

The alt package contains functions and types for altering values. It includes functions for: 

Decompose() a value into simple types of bool, int64, float64, string,
	time.Time, []any and map[string]any.

Recompose() takes simple data type and converts it back into a complex type.

Alter() is the same as decompose except it alters the value in place.

Generify() converts a simple value into a gen.Node.

Asm

The asm package provides a means of building JSON or the corresponding simple types based on a JSON script represented by the Plan type.

Cmd oj

The oj command is a general purpose tool for processing JSON documents. Features include reformatting JSON, colorizing JSON, extracting parts of a JSON document, and filtering. JSONPath is used for both extracting and filtering.

Index

Constants

View Source 
const (
	// NumConvNone leaves a json.Number as is.
	NumConvNone = NumConvMethod(0)

	// NumConvFloat64 convert json.Number to a float64.
	NumConvFloat64 = NumConvMethod('f')

	// NumConvString convert json.Number to a string
	NumConvString = NumConvMethod('s')
)
View Source 
const (
	// Normal is the Normal ANSI encoding sequence.
	Normal = "\x1b[m"
	// Black is the Black ANSI encoding sequence.
	Black = "\x1b[30m"
	// Red is the Red ANSI encoding sequence.
	Red = "\x1b[31m"
	// Green is the Green ANSI encoding sequence.
	Green = "\x1b[32m"
	// Yellow is the Yellow ANSI encoding sequence.
	Yellow = "\x1b[33m"
	// Blue is the Blue ANSI encoding sequence.
	Blue = "\x1b[34m"
	// Magenta is the Magenta ANSI encoding sequence.
	Magenta = "\x1b[35m"
	// Cyan is the Cyan ANSI encoding sequence.
	Cyan = "\x1b[36m"
	// White is the White ANSI encoding sequence.
	White = "\x1b[37m"
	// Gray is the Gray ANSI encoding sequence.
	Gray = "\x1b[90m"
	// BrightRed is the BrightRed ANSI encoding sequence.
	BrightRed = "\x1b[91m"
	// BrightGreen is the BrightGreen ANSI encoding sequence.
	BrightGreen = "\x1b[92m"
	// BrightYellow is the BrightYellow ANSI encoding sequence.
	BrightYellow = "\x1b[93m"
	// BrightBlue is the BrightBlue ANSI encoding sequence.
	BrightBlue = "\x1b[94m"
	// BrightMagenta is the BrightMagenta ANSI encoding sequence.
	BrightMagenta = "\x1b[95m"
	// BrightCyan is the BrightCyan ANSI encoding sequence.
	BrightCyan = "\x1b[96m"
	// BrightWhite is the BrightWhite ANSI encoding sequence.
	BrightWhite = "\x1b[97m"

	// BytesAsString indicates []byte should be encoded as a string.
	BytesAsString = iota
	// BytesAsBase64 indicates []byte should be encoded as base64.
	BytesAsBase64
	// BytesAsArray indicates []byte should be encoded as an array if integers.
	BytesAsArray

	// MaskByTag is the mask for byTag fields.
	MaskByTag = byte(0x10)
	// MaskExact is the mask for Exact fields.
	MaskExact = byte(0x08) // exact key vs lowwer case first letter
	// MaskPretty is the mask for Pretty fields.
	MaskPretty = byte(0x04)
	// MaskNested is the mask for Nested fields.
	MaskNested = byte(0x02)
	// MaskSen is the mask for Sen fields.
	MaskSen = byte(0x01)
	// MaskSet is the mask for Set fields.
	MaskSet = byte(0x20)
	// MaskIndex is the mask for an index that has been set up.
	MaskIndex = byte(0x1f)
)

Variables

View Source 
var (
	// TimeRFC3339Converter converts strings matching time.RFC3339Nano,
	// time.RFC3339, or 2006-01-02 to time.Time.
	TimeRFC3339Converter = Converter{
		String: []func(val string) (any, bool){
			func(val string) (any, bool) {
				if 20 <= len(val) && len(val) <= 35 {
					for _, layout := range []string{time.RFC3339Nano, time.RFC3339} {
						if t, err := time.ParseInLocation(layout, val, time.UTC); err == nil {
							return t, true
						}
					}
				} else if len(val) == 10 {
					if t, err := time.ParseInLocation("2006-01-02", val, time.UTC); err == nil {
						return t, true
					}
				}
				return val, false
			},
		},
	}

	// TimeNanoConverter converts large integers, 946684800000000000
	// (2000-01-01) and above to time.Time.
	TimeNanoConverter = Converter{
		Int: []func(val int64) (any, bool){
			func(val int64) (any, bool) {
				if 946684800000000000 <= val {
					return time.Unix(0, val), true
				}
				return val, false
			},
		},
	}

	// MongoConverter convert maps with one member when the member key is
	// $numberLong, $date, $numberDecimal, or $oid and the value and the
	// member value is a string. These patterns are found in mongodb JSON
	// exports.
	MongoConverter = Converter{
		Map: []func(val map[string]any) (any, bool){
			func(val map[string]any) (any, bool) {
				if len(val) != 1 {
					return val, false
				}
				for k, v := range val {
					s, ok := v.(string)
					if !ok {
						break
					}
					switch k {
					case "$numberLong":
						if i, err := strconv.ParseInt(s, 10, 64); err == nil {
							return i, true
						}
					case "$date":
						if t, err := time.ParseInLocation("2006-01-02T15:04:05.999Z07:00", s, time.UTC); err == nil {
							return t, true
						}
					case "$numberDecimal":
						if f, err := strconv.ParseFloat(s, 64); err == nil {
							return f, true
						}
					case "$oid":
						return s, true
					}
				}
				return val, false
			},
		},
	}
)
View Source 
var (
	// DefaultOptions default options that can be set as desired.
	DefaultOptions = Options{
		InitSize:    256,
		SyntaxColor: Normal,
		KeyColor:    Blue,
		NullColor:   Red,
		BoolColor:   Yellow,
		NumberColor: Cyan,
		StringColor: Green,
		TimeColor:   Magenta,
		HTMLUnsafe:  true,
		WriteLimit:  1024,
	}

	// BrightOptions encoding options for color encoding.
	BrightOptions = Options{
		InitSize:    256,
		SyntaxColor: Normal,
		KeyColor:    BrightBlue,
		NullColor:   BrightRed,
		BoolColor:   BrightYellow,
		NumberColor: BrightCyan,
		StringColor: BrightGreen,
		TimeColor:   BrightMagenta,
		WriteLimit:  1024,
	}

	// GoOptions are the options closest to the go json package.
	GoOptions = Options{
		InitSize:     256,
		SyntaxColor:  Normal,
		KeyColor:     Blue,
		NullColor:    Red,
		BoolColor:    Yellow,
		NumberColor:  Cyan,
		StringColor:  Green,
		TimeColor:    Magenta,
		CreateKey:    "",
		FullTypePath: false,
		OmitNil:      false,
		OmitEmpty:    false,
		UseTags:      true,
		KeyExact:     true,
		NestEmbed:    false,
		BytesAs:      BytesAsBase64,
		WriteLimit:   1024,
	}

	// HTMLOptions defines color options for generating colored HTML. The
	// encoding is suitable for use in a <pre> element.
	HTMLOptions = Options{
		InitSize:    256,
		SyntaxColor: "<span>",
		KeyColor:    `<span style="color:#44f">`,
		NullColor:   `<span style="color:red">`,
		BoolColor:   `<span style="color:#a40">`,
		NumberColor: `<span style="color:#04a">`,
		StringColor: `<span style="color:green">`,
		TimeColor:   `<span style="color:#f0f">`,
		NoColor:     "</span>",
		HTMLUnsafe:  false,
		WriteLimit:  1024,
	}
)
View Source 
var DefaultNumConvMethod = NumConvNone

DefaultNumConvMethod is the default NumConvMethod for parsing and recompose.

View Source 
var ErrorWithStack = false

ErrorWithStack if true the Error() call will include the stack.

Functions

func AppendJSONString 

func AppendJSONString(buf []byte, s string, htmlSafe bool) []byte

AppendJSONString appends a JSON encoding of a string to the provided byte slice.

func AppendSENString 

func AppendSENString(buf []byte, s string, htmlSafe bool) []byte

AppendSENString appends a SEN encoding of a string to the provided byte slice.

func Convert 

func Convert(v any, funcs ...any) any

Convert a value according to the conversion functions provided. If the value is a map or slice and not converted itself the provided value will remain the same but will be modified if any of it's members are converted.

Types

type Converter 

type Converter struct {
	// Int are a slice of functions to match and convert Ints.
	Int []func(val int64) (any, bool)

	// Float are a slice of functions to match and convert Floats.
	Float []func(val float64) (any, bool)

	// String are a slice of functions to match and convert Strings.
	String []func(val string) (any, bool)

	// Map are a slice of functions to match and convert Maps.
	Map []func(val map[string]any) (any, bool)

	// Array are a slice of functions to match and convert Arrays.
	Array []func(val []any) (any, bool)
}

Converter types are used to convert data element to alternate values. Common uses are to match a pattern such as strings representing dates to time.Time.

func (*Converter) Convert 

func (c *Converter) Convert(v any) any

Convert a value according to the conversion functions of the converter. If the value is a map or slice and not converted itself the provided value will remain the same but will be modified if any of it's members are converted.

type Error 

type Error struct {
	// contains filtered or unexported fields
}

Error struct to hold an error message and a stack trace.

func NewError 

func NewError(r any) *Error

NewError creates a new Error instance, capturing the stack when created.

func (*Error) Error 

func (err *Error) Error() string

Error returns a string representation of the instance.

func (*Error) Stack 

func (err *Error) Stack() []byte

Stack returns the stack.

type NumConvMethod added in v1.19.0 

type NumConvMethod byte

NumConvMethod specifies a json.Number conversion method. It is used by the parsers and the recomposer to convert a json.Number to either a float64 or to a string if used.

type Options 

type Options struct {

	// Indent for the output.
	Indent int

	// Tab if true will indent using tabs and ignore the Indent member.
	Tab bool

	// Sort object members if true.
	Sort bool

	// OmitNil skips the writing of nil values in an object.
	OmitNil bool

	// OmitEmpty skips the writing of empty string, slices, maps, and zero
	// values although maps with all empty members will not be skipped on
	// writing but will be with alt.Decompose and alter.
	OmitEmpty bool

	// InitSize is the initial buffer size.
	InitSize int

	// WriteLimit is the size of the buffer that will trigger a write when
	// using a writer.
	WriteLimit int

	// TimeFormat defines how time is encoded. Options are to use a
	// time. layout string format such as time.RFC3339Nano, "second" for a
	// decimal representation, "nano" for a an integer. For decompose setting
	// to "time" will leave it unchanged.
	TimeFormat string

	// TimeWrap if not empty encoded time as an object with a single member. For
	// example if set to "@" then and TimeFormat is RFC3339Nano then the encoded
	// time will look like '{"@":"2020-04-12T16:34:04.123456789Z"}'
	TimeWrap string

	// TimeMap if true will encode time as a map with a create key and a
	// 'value' member formatted according to the TimeFormat options.
	TimeMap bool

	// CreateKey if set is the key to use when encoding objects that can later
	// be reconstituted with an Unmarshall call. This is only use when writing
	// simple types where one of the object in an array or map is not a
	// Simplifier. Reflection is used to encode all public members of the
	// object if possible. For example, is CreateKey is set to "type" this
	// might be the encoding.
	//
	//   { "type": "MyType", "a": 3, "b": true }
	//
	CreateKey string

	// NoReflect if true does not use reflection to encode an object. This is
	// only considered if the CreateKey is empty.
	NoReflect bool

	// FullTypePath if true includes the full type name and path when used
	// with the CreateKey.
	FullTypePath bool

	// Color if true will colorize the output.
	Color bool

	// SyntaxColor is the color for syntax in the JSON output.
	SyntaxColor string

	// KeyColor is the color for a key in the JSON output.
	KeyColor string

	// NullColor is the color for a null in the JSON output.
	NullColor string

	// BoolColor is the color for a bool in the JSON output.
	BoolColor string

	// NumberColor is the color for a number in the JSON output.
	NumberColor string

	// StringColor is the color for a string in the JSON output.
	StringColor string

	// TimeColor is the color for a time.Time in the JSON output.
	TimeColor string

	// NoColor turns the color off.
	NoColor string

	// UseTags if true will use the json annotation tags when marhsalling,
	// writing, or decomposing an struct. If no tag is present then the
	// KeyExact flag is referenced to determine the key.
	UseTags bool

	// KeyExact if true will use the exact field name for an encoded struct
	// field. If false the key style most often seen in JSON files where the
	// first character of the object keys is lowercase.
	KeyExact bool

	// HTMLUnsafe if true turns off escaping of &, <, and >.
	HTMLUnsafe bool

	// NestEmbed if true will generate an element for each anonymous embedded
	// field.
	NestEmbed bool

	// BytesAs indicates how []byte fields should be encoded. Choices are
	// BytesAsString, BytesAsBase64 (the go json package default), or
	// BytesAsArray.
	BytesAs int

	// Converter to use when decomposing or altering if non nil. The Converter
	// type includes more details.
	Converter *Converter

	// FloatFormat is the fmt.Printf formatting verb and options. The default
	// is "%g".
	FloatFormat string
}

Options for writing data to JSON.

func (*Options) AppendTime 

func (o *Options) AppendTime(buf []byte, t time.Time, sen bool) []byte

AppendTime appends a time string to the buffer.

func (*Options) DecomposeTime 

func (o *Options) DecomposeTime(t time.Time) (v any)

DecomposeTime encodes time in the format specified by the settings of the options.

Source Files

View all Source files

Directories

Path Synopsis
Package alt contains functions and types for altering values.
 Package alt contains functions and types for altering values.
Package asm provides a means of building JSON or simple types using JSON encoded scripts.
 Package asm provides a means of building JSON or simple types using JSON encoded scripts.
cmd
benchmarks
Package main is the main package.
 Package main is the main package.
oj
Command oj allows to filter JSON files.
 Command oj allows to filter JSON files.
Package gen provides type safe generic types.
 Package gen provides type safe generic types.
Package jp provides JSONPath implementation that operations on simple go types, generic (gen package), and public struct with public members.
 Package jp provides JSONPath implementation that operations on simple go types, generic (gen package), and public struct with public members.
Package oj contains functions and types to support building simple types where simple types are:
 Package oj contains functions and types to support building simple types where simple types are:
Package pretty contains functions and types to support encoding to JSON and SEN formats is a more pleasing arrangement that either flat or simply indented formats.
 Package pretty contains functions and types to support encoding to JSON and SEN formats is a more pleasing arrangement that either flat or simply indented formats.
Package sen contains the SEN parsers and writers.
 Package sen contains the SEN parsers and writers.
Package tt is a simple test package for the ojg package.
 Package tt is a simple test package for the ojg package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.