iterjson

package module
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 12, 2024 License: MIT Imports: 11 Imported by: 0

README

gopherz

ezpkg.io/iter.json

PkgGoDev GitHub License version

Package iter.json is JSON parser and transformer in Go. The Parse() function returns an iterator over the JSON object, which can be used to traverse the JSON object. And Builder can be used to construct a JSON object. Together, they provide a powerful way to iterate and manipulate your JSON data with Go iterators.

Installation

go get -u ezpkg.io/iter.json@v0.2.2

Examples

Given an example alice.json file:

{
  "name": "Alice",
  "age": 24,
  "scores": [9, 10, 8],
  "address": {
    "city": "The Sun",
    "zip": 10101
  }
}

You can query and manipulate the JSON object in various ways:

1. Iterating JSON:

Use for range Parse() to iterate over a JSON data, then print the path, key, token, and level of each item. See examples/01.iter.

package main

import (
    "fmt"

    "ezpkg.io/errorz"
    iterjson "ezpkg.io/iter.json"
)

func main() {
    data := `{"name": "Alice", "age": 24, "scores": [9, 10, 8], "address": {"city": "The Sun", "zip": 10101}}`

    // 🎄Example: iterate over json
    fmt.Printf("| %12v | %10v | %10v |%v|\n", "PATH", "KEY", "TOKEN", "LVL")
    fmt.Println("| ------------ | ---------- | ---------- | - |")
    for item, err := range iterjson.Parse([]byte(data)) {
        errorz.MustZ(err)

        fmt.Printf("| %12v | %10v | %10v | %v |\n", item.GetPathString(), item.Key, item.Token, item.Level)
    }
}

The code will output:

|         PATH |        KEY |      TOKEN |LVL|
| ------------ | ---------- | ---------- | - |
|              |            |          { | 0 |
|         name |     "name" |    "Alice" | 1 |
|          age |      "age" |         24 | 1 |
|       scores |   "scores" |          [ | 1 |
|     scores.0 |            |          9 | 2 |
|     scores.1 |            |         10 | 2 |
|     scores.2 |            |          8 | 2 |
|       scores |            |          ] | 1 |
|      address |  "address" |          { | 1 |
| address.city |     "city" |  "The Sun" | 2 |
|  address.zip |      "zip" |      10101 | 2 |
|      address |            |          } | 1 |
|              |            |          } | 0 |
2. Building JSON:

Use Builder to build a JSON data. It accepts optional arguments for indentation. See examples/02.builder.

b := iterjson.NewBuilder("", "    ")
// open an object
b.Add("", iterjson.TokenObjectOpen)

// add a few fields
b.Add("name", "Alice")
b.Add("age", 22)
b.Add("email", "alice@example.com")
b.Add("phone", "(+84) 123-456-789")

// open an array
b.Add("languages", iterjson.TokenArrayOpen)
b.Add("", "English")
b.Add("", "Vietnamese")
b.Add("", iterjson.TokenArrayClose)
// close the array

// accept any type that can marshal to json
b.Add("address", Address{
    HouseNumber: 42,
    Street:      "Ly Thuong Kiet",
    City:        "Ha Noi",
    Country:     "Vietnam",
})

// accept []byte as raw json
b.Add("pets", []byte(`[{"type":"cat","name":"Kitty","age":2},{"type":"dog","name":"Yummy","age":3}]`))

// close the object
b.Add("", iterjson.TokenObjectClose)

out := errorz.Must(b.Bytes())
fmt.Printf("\n--- build json ---\n%s\n", out)

Which will output the JSON with indentation:

{
    "name": "Alice",
    "age": 22,
    "email": "alice@example.com",
    "phone": "(+84) 123-456-789",
    "languages": [
        "English",
        "Vietnamese"
    ],
    "address": {"house_number":42,"street":"Ly Thuong Kiet","city":"Ha Noi","country":"Vietnam"},
    "pets": [
        {
            "type": "cat",
            "name": "Kitty",
            "age": 2
        },
        {
            "type": "dog",
            "name": "Yummy",
            "age": 3
        }
    ]
}
3. Formatting JSON:

You can reconstruct or format a JSON data by sending its key and values to a Builder. See examples/03.reformat.

{
    // 🐝Example: minify json
    b := iterjson.NewBuilder("", "")
    for item, err := range iterjson.Parse(data) {
        errorz.MustZ(err)
        b.AddRaw(item.Key, item.Token)
    }
    out := errorz.Must(b.Bytes())
    fmt.Printf("\n--- minify ---\n%s\n----------\n", out)
}
{
    // 🦋Example: format json
    b := iterjson.NewBuilder("👉   ", "\t")
    for item, err := range iterjson.Parse(data) {
        errorz.MustZ(err)
        b.AddRaw(item.Key, item.Token)
    }
    out := errorz.Must(b.Bytes())
    fmt.Printf("\n--- reformat ---\n%s\n----------\n", out)
}

The first example minifies the JSON while the second example formats it with prefix "👉" on each line.

--- minify ---
{"name":"Alice","age":24,"scores":[9,10,8],"address":{"city":"The Sun","zip":10101}}
----------

--- reformat ---
👉   {
👉       "name": "Alice",
👉       "age": 24,
👉       "scores": [
👉           9,
👉           10,
👉           8
👉       ],
👉       "address": {
👉           "city": "The Sun",
👉           "zip": 10101
👉       }
👉   }
----------
4. Adding line numbers

In this example, we add line numbers to the JSON output, by adding a b.WriteNewline() before the fmt.Fprintf() call. See examples/04.line_number.

// 🐞Example: print with line number
i := 0
b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)
    b.WriteNewline(item.Token.Type())

    // 👉 add line number
    fmt.Fprintf(b, "%3d    ", i)
    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- line number ---\n%s\n----------\n", out)

This will output:

  1    {
  2        "name": "Alice",
  3        "age": 24,
  4        "scores": [
  5            9,
  6            10,
  7            8
  8        ],
  9        "address": {
 10            "city": "The Sun",
 11            "zip": 10101
 12        }
 13    }
5. Adding comments

By putting a fmt.Fprintf(comment) between b.WriteComma() and b.WriteNewline(), you can add a comment to the end of each line. See examples/05.comment.

i, newlineIdx, maxIdx := 0, 0, 30
b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    b.WriteComma(item.Token.Type())

    // 👉 add comment
    if i > 0 {
        length := b.Len() - newlineIdx
        fmt.Fprint(b, strings.Repeat(" ", maxIdx-length))
        fmt.Fprintf(b, "// %2d", i)
    }
    i++

    b.WriteNewline(item.Token.Type())
    newlineIdx = b.Len() // save the newline index

    b.Add(item.Key, item.Token)
}
length := b.Len() - newlineIdx
fmt.Fprint(b, strings.Repeat(" ", maxIdx-length))
fmt.Fprintf(b, "// %2d", i)

out := errorz.Must(b.Bytes())
fmt.Printf("\n--- comment ---\n%s\n----------\n", out)

This will output:

{                             //  1
    "name": "Alice",          //  2
    "age": 24,                //  3
    "scores": [               //  4
        9,                    //  5
        10,                   //  6
        8                     //  7
    ],                        //  8
    "address": {              //  9
        "city": "The Sun",    // 10
        "zip": 10101          // 11
    }                         // 12
}                             // 13
6. Filtering JSON and extracting values

There are item.GetPathString() and item.GetRawPath() to get the path of the current item. You can use them to filter the JSON data. See examples/06.filter_print.

Example with item.GetPathString() and regexp:

fmt.Printf("\n--- filter: GetPathString() ---\n")
i := 0
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)

    path := item.GetPathString()
    switch {
    case path == "name",
        strings.Contains(path, "address"):
        // continue
    default:
        continue
    }

    // 👉 print with line number
    fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath())
}

Example with item.GetRawPath() and path.Match():

fmt.Printf("\n--- filter: GetRawPath() ---\n")
i := 0
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)

    path := item.GetRawPath()
    switch {
    case path.Match("name"),
        path.Contains("address"):
        // continue
    default:
        continue
    }

    // 👉 print with line number
    fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath())
}

Both examples will output:

 2              "Alice" . name
 9                    { . address
10            "The Sun" . address.city
11                10101 . address.zip
12                    } . address
7. Filtering JSON and returning a new JSON

By combining the Builder with the option SetSkipEmptyStructures(false) and the filtering logic, you can filter the JSON data and return a new JSON. See examples/07.filter_json

// 🦁Example: filter and output json
b := iterjson.NewBuilder("", "    ")
b.SetSkipEmptyStructures(true) // 👉 skip empty [] or {}
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    if item.Token.IsOpen() || item.Token.IsClose() {
        b.Add(item.Key, item.Token)
        continue
    }

    path := item.GetPathString()
    switch {
    case path == "name",
        strings.Contains(path, "address"):
        // continue
    default:
        continue
    }

    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- filter: output json ---\n%s\n----------\n", out)

This example will return a new JSON with only the filtered fields:

{
    "name": "Alice",
    "address": {
        "city": "The Sun",
        "zip": 10101
    }
}
8. Editing values

This is an example for editing values in a JSON data. Assume that we are using number ids for our API. The ids are too big and JavaScript can't handle them. We need to convert them to strings. See examples/08.number_id and order.json.

Iterate over the JSON data, find all _id fields and convert the number ids to strings:

b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    key, _ := item.GetRawPath().Last().ObjectKey()
    if strings.HasSuffix(key, "_id") {
        id, err0 := item.Token.GetInt()
        if err0 == nil {
            b.Add(item.Key, strconv.Itoa(id))
            continue
        }
    }
    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- convert number id ---\n%s\n----------\n", out)

This will add quotes to the number ids:

{
    "order_id": "12345678901234",
    "number": 12,
    "customer_id": "12345678905678",
    "items": [
        {
            "item_id": "12345678901042",
            "quantity": 1,
            "price": 123.45
        },
        {
            "item_id": "12345678901098",
            "quantity": 2,
            "price": 234.56
        }
    ]
}

About ezpkg.io

As I work on various Go projects, I often find myself creating utility functions, extending existing packages, or developing packages to solve specific problems. Moving from one project to another, I usually have to copy or rewrite these solutions. So I created this repository to have all these utilities and packages in one place. Hopefully, you'll find them useful as well.

For more information, see the main repository.

Author

Oliver Nguyen  github

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrTokenInvalid = errors.New("invalid token")
	ErrTokenEmpty   = errors.New("invalid empty token")
	ErrTokenString  = errors.New("invalid string token")
	ErrTokenNumber  = errors.New("invalid number token")
	ErrNumberNotInt = errors.New("number is not an integer")
	ErrTokenBool    = errors.New("invalid boolean token")
	ErrTokenType    = errors.New("invalid token type")
)

Functions

func Parse 

func Parse(in []byte) iter.Seq2[Item, error]

func Reconstruct 

func Reconstruct(in []byte) ([]byte, error)

Reconstruct is an example of how to reconstruct a JSON from Parse().

func Reformat 

func Reformat(in []byte, prefix, indent string) ([]byte, error)

Reformat is an example of how to reconstruct a JSON from Parse(), and format with indentation.

func Scan 

func Scan(in []byte) iter.Seq2[RawToken, error]

func ShouldAddComma 

func ShouldAddComma(lastToken, nextToken TokenType) bool

ShouldAddComma returns true if a comma should be added before the next token.

Types

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

func NewBuilder 

func NewBuilder(prefix, indent string) *Builder

NewBuilder creates a new Builder. It's optional to set the prefix and indent. A zero Builder is valid.

func (*Builder) Add 

func (b *Builder) Add(key any, value any)

Add adds a key and value to the builder. It will add a comma if needed.

func (*Builder) AddRaw 

func (b *Builder) AddRaw(key, token RawToken)

AddRaw adds a key and token to the builder. It will add a comma if needed.

func (*Builder) Bytes 

func (b *Builder) Bytes() ([]byte, error)

Bytes returns the bytes of the builder with an error if any.

func (*Builder) Len 

func (b *Builder) Len() int

Len returns the number of bytes written.

func (*Builder) SetSkipEmptyStructures 

func (b *Builder) SetSkipEmptyStructures(skip bool)

SetSkipEmptyStructures makes the Builder ignores empty array `[]` and empty object `{}`. Default to false.

func (*Builder) ShouldAddComma 

func (b *Builder) ShouldAddComma(next TokenType) bool

ShouldAddComma returns true if a comma should be added before the next token.

func (*Builder) Write 

func (b *Builder) Write(p []byte) (n int, err error)

func (*Builder) WriteComma 

func (b *Builder) WriteComma(next TokenType)

WriteCommand writes a comma if needed.

func (*Builder) WriteIndent 

func (b *Builder) WriteIndent()

WriteIndent writes the indentation if needed.

func (*Builder) WriteNewline 

func (b *Builder) WriteNewline(next TokenType)

WriteNewline writes a newline if needed.

type Item 

type Item struct {
	Level int      // level of indentation
	Index int      // index in the parent array or object
	Key   RawToken // optional object "key"
	Token RawToken // [ or { or } or ] or , or value
	// contains filtered or unexported fields
}

func (Item) Format 

func (x Item) Format(f fmt.State, c rune)

func (Item) GetAltPathString 

func (x Item) GetAltPathString() string

GetAltPathString returns the path of the item as a string `[0].key[1]`.

func (Item) GetPath 

func (x Item) GetPath() Path

GetPath returns the path of the item as a slice of values. The values are the keys of objects (string) and the indexes of arrays (int).

func (Item) GetPathString 

func (x Item) GetPathString() string

GetPathString returns the path of the item as a string "0.key.1".

func (Item) GetRawPath 

func (x Item) GetRawPath() RawPath

GetRawPath returns the path of the item as a slice of PathItem. IMPORTANT: The result slice should not be modified.

func (Item) GetTokenValue 

func (x Item) GetTokenValue() (any, error)

func (Item) GetValue 

func (x Item) GetValue() (any, error)

func (Item) IsArrayValue 

func (x Item) IsArrayValue() bool

func (Item) IsObjectValue 

func (x Item) IsObjectValue() bool

func (Item) String 

func (x Item) String() string

type Path 

type Path []any

Path is a slice of values. The values are the keys of objects (string) and the indexes of arrays (int).

func (Path) Format 

func (p Path) Format(f fmt.State, c rune)

Path returns the path of the item as a string. Default to 0.key.1 or "%+v" to format as [0]."key"[1]

func (Path) String 

func (p Path) String() string

Path returns the path of the item as a string. Default to 0.key.1 or "%+v" to format as [0]."key"[1]

type PathItem 

type PathItem struct {
	Index int      // array index or object index
	Key   RawToken // object key
	Token RawToken // [ or { or } or ]
}

func NewPathItem 

func NewPathItem(x any) (pi PathItem, err error)

NewPathItem returns a new PathItem. Use int for array index and string for object key. Keys should be quoted for better performance (optional).

func (PathItem) ArrayIndex 

func (p PathItem) ArrayIndex() (int, bool)

ArrayIndex returns the array index of the path item. It returns the index as an int if the item is inside an array.

func (PathItem) Format 

func (p PathItem) Format(f fmt.State, c rune)

Format formats the path item as a string. Use "%+v" to format as "[0]" for array, ".key" for object.

func (PathItem) IsArray 

func (p PathItem) IsArray() bool

IsArray returns true if the path item is inside an array.

func (PathItem) IsArrayIndex 

func (p PathItem) IsArrayIndex(idx int) bool

IsArrayIndex returns true if the path item is inside an array and the index matches the given index.

func (PathItem) IsObject 

func (p PathItem) IsObject() bool

IsObject returns true if the path item is inside an object.

func (PathItem) IsObjectKey 

func (p PathItem) IsObjectKey(key string) bool

IsObjectKey returns true if the path item is inside an object and the key matches the given key.

func (PathItem) IsObjectRawKey 

func (p PathItem) IsObjectRawKey(rawKey string) bool

IsObjectRawKey returns true if the path item is inside an object and the key matches the given unquote key.

func (PathItem) Match 

func (p PathItem) Match(x any) bool

Match returns true if the path item matches the given value. The value must be an int for array index or a string|[]byte for object key.

func (PathItem) ObjectKey 

func (p PathItem) ObjectKey() (string, bool)

ObjectKey returns the object key of the path item. It returns the key as a string if the item is inside an object.

func (PathItem) String 

func (p PathItem) String() string

String returns the string representation of the path item. "0" for array, "key" for object.

func (PathItem) Value 

func (p PathItem) Value() any

Value returns the value of the path item. If the item is inside an array, it returns the index. If the item is inside an object, it returns the key.

type RawPath 

type RawPath []PathItem

func NewRawPath 

func NewRawPath(parts ...any) (path RawPath, err error)

NewRawPath returns a new RawPath. Use int for array index and string for object key. Keys should be quoted for better performance (optional).

func (RawPath) Contains 

func (p RawPath) Contains(X ...any) bool

ContainsRaw returns true if the path contains the given path, by raw values (int for array index, string for object key). Examples: 

path.Contains([]any{1, "key", 2})

func (RawPath) Format 

func (p RawPath) Format(f fmt.State, c rune)

Format formats the path as a string. Default to 0.key.1 or "%+v" to format as [0]."key"[1]

func (RawPath) Last 

func (p RawPath) Last() PathItem

Last returns the last item of the path.

func (RawPath) Match 

func (p RawPath) Match(X ...any) bool

Match returns true if the path is equal to the given path, by raw values (int for array index, string for object key). Examples: 

path.Match([]any{1, "key", 2})

func (RawPath) String 

func (p RawPath) String() string

String returns the path of the item as a string. Default to 0.key.1 or "%+v" to format as [0]."key"[1]

type RawToken 

type RawToken struct {
	// contains filtered or unexported fields
}

RawToken represents a raw token from the scanner.

func BoolToken 

func BoolToken(b bool) RawToken

BoolToken returns a boolean token.

func IntToken 

func IntToken(n int) RawToken

IntToken returns a number token.

func MustRawToken 

func MustRawToken(raw []byte) RawToken

MustRawToken returns a new raw token from the raw bytes. Panic if error.

func NewRawToken 

func NewRawToken(raw []byte) (RawToken, error)

NewRawToken returns a new raw token from the raw bytes.

func NextToken 

func NextToken(in []byte) (token RawToken, remain []byte, err error)

func NumberToken 

func NumberToken(f float64) RawToken

NumberToken returns a number token. For NaN and Inf, fallback to 0.

func StringToken 

func StringToken(s string) RawToken

StringToken returns a string token.

func (RawToken) Bytes 

func (r RawToken) Bytes() []byte

Bytes returns the raw bytes value of the token.

func (RawToken) Equal 

func (r RawToken) Equal(other RawToken) bool

Equal returns true if the token is equal to the other token.

func (RawToken) GetBool 

func (r RawToken) GetBool() (bool, error)

GetBool returns the boolean value of the token.

func (RawToken) GetInt 

func (r RawToken) GetInt() (int, error)

GetInt returns the integer value of the token.

func (RawToken) GetNumber 

func (r RawToken) GetNumber() (float64, error)

GetNumber returns the number value of the token.

func (RawToken) GetString 

func (r RawToken) GetString() (string, error)

GetString returns the unquoted string value of the token. https://datatracker.ietf.org/doc/html/rfc8259#section-7

func (RawToken) GetValue 

func (r RawToken) GetValue() (any, error)

GetValue returns the value of the token as an any.

func (RawToken) IsClose 

func (r RawToken) IsClose() bool

IsClose returns true if the token is a close token ']' or '}'.

func (RawToken) IsOpen 

func (r RawToken) IsOpen() bool

IsOpen returns true if the token is an open token '[' or '{'.

func (RawToken) IsValue 

func (r RawToken) IsValue() bool

IsValue returns true if the token is a value.

func (RawToken) IsZero 

func (r RawToken) IsZero() bool

IsZero returns true if the token is zero.

func (RawToken) Raw 

func (r RawToken) Raw() []byte

Raw returns the raw bytes value of the token.

func (RawToken) String 

func (r RawToken) String() string

String returns the raw string value of the token. Use ToString() for unquoted strings.

func (RawToken) Type 

func (r RawToken) Type() TokenType

Type returns the type of the token.

type TokenType 

type TokenType byte

TokenType represents the type of a JSON token. 

const (
	TokenNull        TokenType = 'n'
	TokenTrue        TokenType = 't'
	TokenFalse       TokenType = 'f'
	TokenNumber      TokenType = '0'
	TokenString      TokenType = '"'
	TokenObjectOpen  TokenType = '{'
	TokenObjectClose TokenType = '}'
	TokenArrayOpen   TokenType = '['
	TokenArrayClose  TokenType = ']'
	TokenComma       TokenType = ','
	TokenColon       TokenType = ':'
)

func (TokenType) IsClose 

func (t TokenType) IsClose() bool

IsClose returns true if the token is a close token ']' or '}'.

func (TokenType) IsOpen 

func (t TokenType) IsOpen() bool

IsOpen returns true if the token is an open token '[' or '{'.

func (TokenType) IsValue 

func (t TokenType) IsValue() bool

IsValue returns true if the token is a value: null, boolean, number, string.

func (TokenType) New 

func (t TokenType) New() RawToken

New returns a new raw token of the type. For number and string, use NumberToken() and StringToken().

func (TokenType) String 

func (t TokenType) String() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.