README
¶
jsonconv
Description
A Golang library and cmd for flattening JSON and converting JSON to CSV.
With jsonconv, you can:
- Flatten a JSON object which contains deeply nested JSON object and JSON array.
- Convert a JSON object or JSON array to CSV data in a flexible way.
- Use jsonconv cmd (built with cobra) as convenient tool.
Installation
First, use go get to install the latest version of the library:
go get -u github.com/tuan78/jsonconv@latest
Next, include jsonconv in your application:
import "github.com/tuan78/jsonconv"
Usage
Flatten JSON Object
obj := map[string]any{
"a": 1,
"b": 2,
"c": map[string]any{
"d": map[string]any{
"e": 3,
},
},
"f": []int{4, 5, 6},
"g": map[string]any{
"h": "A",
"i": true,
"j": 1,
"k": 1.5,
"l": nil,
},
}
jsonconv.Flatten(obj, nil) // The 'obj' will be modified as result
Result:
map[string]any{
"a": 1,
"b": 2,
"c__d__e": 3,
"f[0]": 4,
"f[1]": 5,
"f[2]": 6,
"g__h": "A",
"g__i": true,
"g__j": 1,
"g__k": 1.5,
"g__l": nil,
}
To customize the flattened data, you can create FlattenOption and put it in 2nd parameter. For example:
obj := map[string]any{
"a": 1,
"b": 2,
"c": map[string]any{
"d": map[string]any{
"e": 3,
},
},
"f": []int{4, 5, 6},
"g": map[string]any{
"h": "A",
"i": true,
"j": 1,
"k": 1.5,
"l": nil,
},
}
opt := &jsonconv.FlattenOption{
Level: 1, // Only flatten the 1st nested JSON object and JSON array
Gap: "|", // Change the gap style between parent object and nested object
SkipArray: true, // Skip JSON array flattening
SkipObject: false, // Skip JSON object flattening. For example, leave it as 'false'
}
jsonconv.Flatten(obj, opt) // The 'obj' will be modified as result
Result:
map[string]any{
"a": 1,
"b": 2,
"c|d": map[string]any{
"e": 3,
},
"f": []int{4, 5, 6},
"g|h": "A",
"g|i": true,
"g|j": 1,
"g|k": 1.5,
"g|l": nil,
}
Convert JSON Object or JSON Array to CSV Data
arr := []map[string]any{
{
"id": "b042ab5c-ca73-4460-b739-96410ea9d3a6",
"user": "Jon Doe",
"score": -100,
"is active": false,
},
{
"id": "ce06f5b1-5721-42c0-91e1-9f72a09c250a",
"user": "Tuấn",
"score": 1.5,
"is active": true,
"nested": map[string]any{
"a": 1,
"b": 2,
},
},
{
"id": "4e01b638-44e5-4079-8043-baabbff21cc8",
"user": "高橋",
"score": 100000000000000000,
"is active": true,
},
}
result := jsonconv.ToCsv(arr, nil)
Result:
[][]string{
{
"id", "is active", "nested", "score", "user"
},
{
"b042ab5c-ca73-4460-b739-96410ea9d3a6", "false", "", "-100", "Jon Doe"
},
{
"ce06f5b1-5721-42c0-91e1-9f72a09c250a", "true", "map[a:1 b:2]", "1.5", "Tuấn"
},
{
"4e01b638-44e5-4079-8043-baabbff21cc8", "true", "", "100000000000000000", "高橋"
},
}
From the result above, we can see that the header is sorted and JSON object is not flattened.
So to customize the csv data, you can create ToCsvOption and put it in 2nd parameter. For example:
arr := []map[string]any{
{
"id": "b042ab5c-ca73-4460-b739-96410ea9d3a6",
"user": "Jon Doe",
"score": -100,
"is active": false,
},
{
"id": "ce06f5b1-5721-42c0-91e1-9f72a09c250a",
"user": "Tuấn",
"score": 1.5,
"is active": true,
"nested": map[string]any{
"a": 1,
"b": 2,
},
},
{
"id": "4e01b638-44e5-4079-8043-baabbff21cc8",
"user": "高橋",
"score": 100000000000000000,
"is active": true,
},
}
opt := &jsonconv.ToCsvOption{
FlattenOption: jsonconv.DefaultFlattenOption, // Let's try default flatten option this time
BaseHeaders: []string{"id", "user"}, // To keep 'id' column and 'user' column before the rest
}
result := jsonconv.ToCsv(arr, opt)
Result:
[][]string{
{
"id", "user", "is active", "nested_a", "nested_b", "score"
},
{
"b042ab5c-ca73-4460-b739-96410ea9d3a6", "Jon Doe", "false", "", "", "-100"
},
{
"ce06f5b1-5721-42c0-91e1-9f72a09c250a", "Tuấn", "true", "1", "2", "1.5"
},
{
"4e01b638-44e5-4079-8043-baabbff21cc8", "高橋", "true", "", "", "100000000000000000"
},
}
Cmd
To install the latest version of jsonconv cmd, you can use go install command:
go install github.com/tuan78/jsonconv/cmd/jsonconv@latest
Next, to see what jsonconv cmd can do, you can run help for any command. For example:
jsonconv help
jsonconv csv --help
jsonconv flatten --help
Flatten JSON Object or JSON Array
To flatten JSON from JSON file and output fattened JSON file, you just simply run:
jsonconv flatten -i sample.json -o fattened.json
Alternatively, without -i (or --in) and without -o (or --out) in your command, it reads data from Stdin and prints result to Stdout. So you can execute command in your convenient way:
cat sample.json | jsonconv flatten
Convert JSON Object or JSON Array to CSV Data
To convert JSON from JSON file to CSV file, you can run:
jsonconv csv -i sample.json -o converted.csv
Or you want to read from Stdin and print to Stdout. For example:
cat sample.json | jsonconv csv
Notes that the csv command will flatten the json by default. If you don't want JSON flattening feature, you can skip it by using --noft (no flattening) flag. For example:
cat sample.json | jsonconv csv --noft
License
jsonconv is released under the MIT license. See LICENSE
Documentation
¶
Overview ¶
Package jsonconv provides functionality for flattening JSON objects and converting JSON to CSV. It supports deep nested JSON structures and offers flexible configuration options.
Index ¶
Constants ¶
const ( // FlattenLevelUnlimited can be set to FlattenOption.Level for unlimited flattening. FlattenLevelUnlimited = -1 // FlattenLevelNonNested can be set to FlattenOption.Level for non-nested flattening // (equivalent to non-flattening). FlattenLevelNonNested = 0 )
const ( // DefaultFlattenLevel can be set to FlattenOption.Level for default level flattening // (equivalent to FlattenLevelUnlimited). DefaultFlattenLevel = FlattenLevelUnlimited // DefaultFlattenGap can be set to FlattenOption.Gap for default gap flattening. DefaultFlattenGap = "__" )
const CsvComma rune = ','
CsvComma is the default field delimiter for CSV files.
Variables ¶
var DefaultFlattenOption = &FlattenOption{ Level: DefaultFlattenLevel, Gap: DefaultFlattenGap, }
DefaultFlattenOption provides default settings for flattening operations.
Functions ¶
func CreateCsvHeader ¶
CreateCsvHeader creates []string from arr and baseHs. A baseHs is base header that we want to put at the beginning of dynamic header, we can set baseHs to nil if we just want to have dynamic header only.
func Flatten ¶
func Flatten(obj map[string]any, opt *FlattenOption)
Flatten flattens obj with given opt. If opt is nil, it will use opt value from DefaultFlattenOption instead.
Types ¶
type CsvWriter ¶
type CsvWriter struct {
// Field delimiter. Set to ',' (CsvComma) by default in NewCsvWriter
Delimiter rune
// True to use \r\n as the line terminator
UseCRLF bool
// contains filtered or unexported fields
}
A CsvWriter writes records using CSV encoding.
func NewCsvWriter ¶
NewCsvWriter returns a new CsvWriter that writes to w.
type FlattenOption ¶
type FlattenOption struct {
// Level of flattening, it can be FlattenLevelUnlimited,
// FlattenLevelNonNested or an int value in [1..n]
Level int
// A gap between nested JSON and its parent JSON.
// It will be used when merging nested JSON's key with parent JSON's key
Gap string
// Skip Map type (typically JSON Object type) from flattening process
SkipMap bool
// Skip Array type (JSON array, string array, int array, float array, etc.)
// from flattening process
SkipArray bool
}
A FlattenOption is for JSON object flattening.
type JsonReader ¶
type JsonReader struct {
// contains filtered or unexported fields
}
A JsonReader reads and decodes JSON values from an input stream.
func NewJsonReader ¶
func NewJsonReader(r io.ReadSeeker) *JsonReader
NewJsonReader returns a new JsonReader that reads from r.
func (*JsonReader) Read ¶
func (r *JsonReader) Read(v any) error
Read reads the next JSON-encoded value from its input and stores it in the value pointed to by v.
type JsonWriter ¶
type JsonWriter struct {
// EscapeHTML specifies whether problematic HTML characters
// should be escaped inside JSON quoted strings.
// The default behavior is to escape &, <, and > to \u0026, \u003c, and \u003e
// to avoid certain safety problems that can arise when embedding JSON in HTML.
EscapeHTML bool
// contains filtered or unexported fields
}
A JsonWriter writes JSON values to an output stream.
func NewJsonWriter ¶
func NewJsonWriter(w io.Writer) *JsonWriter
NewJsonWriter returns a new JsonWriter that writes to w.
func (*JsonWriter) Write ¶
func (r *JsonWriter) Write(v any) error
Write writes the JSON encoding of v to the stream, followed by a newline character.
type ToCsvOption ¶
type ToCsvOption struct {
// Set it to apply JSON flattening
FlattenOption *FlattenOption
// Base CSV headers used to add before dynamic headers
BaseHeaders []string
}
A ToCsvOption converts a JSON Array to CSV data.
Source Files
Directories