json2toon

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 31, 2026 License: MIT Imports: 11 Imported by: 0

README

JSON2TOON

JSON2TOON

JSON/JSONC/JSONL to TOON (and back) converter library written in Go.

GoDoc TOON Format

Features

  • Bidirectional conversion - Convert JSON to TOON and back
  • Streaming support - Token-based JSON parsing for large files
  • JSONC support - Handles // and /* */ comments before parsing
  • JSONL support - Auto-detects and converts JSON Lines format
  • TOON format - Produces valid TOON output (Format Overview)
  • Configurable - Indent, delimiter, key folding, strict mode
  • CLI tool - j2t command for convenient file/stdin conversion

Installation

go get github.com/martianzhang/json2toon

CLI

# Install
go install github.com/martianzhang/json2toon/cmd/j2t@latest

# Convert file (auto-detects JSON/JSONC/JSONL)
j2t config.json
j2t config.jsonc
j2t data.jsonl

# With options
j2t -indent 4 -delimiter $'\t' -key-folding safe config.json

# Stdin
echo '{"id": 1}' | j2t
CLI Options
Flag Default Description
-o stdout Output file path
-e auto Force encode mode (JSON to TOON)
-d, -r auto Force decode mode (TOON to JSON)
-indent 2 Spaces per indentation level
-delimiter , Table delimiter (,, \t, |)
-key-folding (none) Key folding mode (off, safe)
-strict false Enable strict TOON validation
-stream false Streaming mode for JSONL (line-by-line, memory-efficient)
--stats false Show token count estimates and savings
--expandPaths (none) Path expansion mode on decode (off, safe)
File Extension Detection

The CLI auto-detects operation based on file extension:

  • .json, .jsonc, .jsonl → encode (JSON to TOON)
  • .toon → decode (TOON to JSON)
Practical Tips
# Convert API response for LLM consumption (TOON uses up to 60% fewer tokens than JSON)
curl -s https://api.example.com/data | j2t

# See token savings estimate
curl -s https://api.example.com/data | j2t --stats >/dev/null

# Convert and save to file
curl -s https://api.example.com/users | j2t -o users.toon

# Extract a nested array first with jq, then convert
curl -s https://api.example.com/data | jq '.users' | j2t

# Convert JSONL by merging lines into array first
cat data.jsonl | jq -s '.' | j2t

# Convert each JSONL object individually (keeps objects separate)
cat data.jsonl | jq -c '.' | j2t -stream

# Reverse conversion: TOON to JSON
echo 'id: 123' | j2t -r
# Output: { "id": 123 }

# Convert TOON file to JSON (auto-detected from .toon extension)
j2t config.toon -o config.json

# Output to file
j2t config.json -o config.toon

# Path expansion for lossless round-trip with key folding
j2t compressed.toon --expandPaths safe -o output.json

TOON reduces LLM costs. For tabular data, it cuts token usage by up to 60% vs JSON with equal or better accuracy — see benchmark results for details.

Library Quick Start

package main

import (
	"fmt"
	"github.com/martianzhang/json2toon"
)

func main() {
	json := `{"id": 123, "name": "Ada", "active": true}`
	result, err := json2toon.Convert([]byte(json))
	if err != nil {
		panic(err)
	}
	fmt.Println(string(result))
}

Output:

id: 123
name: Ada
active: true

API Reference

All API details (function signatures, options, examples) are on the godoc page. Key highlights:

  • Basic conversion: Convert, ConvertString, ConvertFile, ConvertFileToWriter
  • Format-specific: ConvertJSONC, ConvertJSONL, ConvertJSONLStream (and ...String / ...WithOptions variants)
  • Auto-detection: ConvertAuto, ConvertAutoFile (detect JSON/JSONC/JSONL automatically)
  • Reverse conversion: ConvertToJSON, ConvertToJSONString, ConvertToJSONFile, ConvertToJSONFromReader
  • Streaming: NewConverter, ConvertJSONLStream on a Converter instance
  • Options: WithIndent, WithDelimiter, WithKeyFolding, WithStrict (encoder); WithDecodeStrict, WithExpandPaths (decoder)

TOON Format

For detailed format specification, see the official TOON documentation.

Makefile Targets

make test      # Run tests
make lint      # Run golangci-lint
make build     # Build CLI
make release   # Cross-compile for Linux/macOS/Windows

Benchmarks

BenchmarkConvertSimple-8          8137 ns/op     6064 B/op      60 allocs/op
BenchmarkConvertNested-8         11511 ns/op     6576 B/op      87 allocs/op
BenchmarkConvertArray-8          69058 ns/op    25496 B/op     834 allocs/op
BenchmarkConvertTabular-8       313393 ns/op    97368 B/op    3328 allocs/op
BenchmarkConvertLargeFile-8     919854 ns/op   360137 B/op   10379 allocs/op
BenchmarkConvertJSONC-8          10348 ns/op     6664 B/op      82 allocs/op
BenchmarkConvertJSONL-8        1720938 ns/op    83922 B/op    1866 allocs/op
BenchmarkConvertJSONLStream-8    228059 ns/op    95323 B/op    2657 allocs/op
BenchmarkConvertAuto-8            9513 ns/op     7264 B/op      96 allocs/op
BenchmarkConvertToJSON-8          8054 ns/op     9678 B/op      40 allocs/op

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Convert 

func Convert(jsonBytes []byte) ([]byte, error)

Convert converts JSON bytes to TOON bytes.

func ConvertAuto 

func ConvertAuto(data []byte) ([]byte, error)

ConvertAuto auto-detects and converts input format (JSON/JSONC/JSONL) to TOON.

func ConvertAutoFile 

func ConvertAutoFile(path string) ([]byte, error)

ConvertAutoFile converts a file with auto-detected format to TOON bytes.

func ConvertAutoString 

func ConvertAutoString(s string) (string, error)

ConvertAutoString converts input string to TOON with auto format detection.

func ConvertAutoWithOptions 

func ConvertAutoWithOptions(data []byte, opts ...ConverterOption) ([]byte, error)

ConvertAutoWithOptions converts to TOON with auto format detection and custom options.

func ConvertFile 

func ConvertFile(path string) ([]byte, error)

ConvertFile converts a JSON file to TOON bytes.

func ConvertFileToWriter 

func ConvertFileToWriter(path string, w io.Writer) error

ConvertFileToWriter converts a JSON file and writes to an io.Writer.

func ConvertFileWithOptions 

func ConvertFileWithOptions(path string, opts ...ConverterOption) ([]byte, error)

ConvertFileWithOptions converts a JSON file with custom options.

func ConvertJSONC 

func ConvertJSONC(jsoncBytes []byte) ([]byte, error)

ConvertJSONC converts JSONC (JSON with comments) to TOON bytes.

func ConvertJSONCString 

func ConvertJSONCString(jsoncStr string) (string, error)

ConvertJSONCString converts a JSONC string to TOON string.

func ConvertJSONCWithOptions 

func ConvertJSONCWithOptions(jsoncBytes []byte, opts ...ConverterOption) ([]byte, error)

ConvertJSONCWithOptions converts JSONC to TOON with custom options.

func ConvertJSONL 

func ConvertJSONL(jsonlBytes []byte) ([]byte, error)

ConvertJSONL converts JSON Lines (JSONL) to TOON bytes.

func ConvertJSONLStream 

func ConvertJSONLStream(jsonlBytes []byte) ([]byte, error)

ConvertJSONLStream converts JSON Lines to TOON bytes using streaming mode. Each line is processed and written immediately, separated by "---". This is more memory-efficient for large JSONL files.

func ConvertJSONLStreamString 

func ConvertJSONLStreamString(jsonlStr string) (string, error)

ConvertJSONLStreamString converts JSON Lines string to TOON using streaming mode.

func ConvertJSONLStreamWithOptions 

func ConvertJSONLStreamWithOptions(jsonlBytes []byte, opts ...ConverterOption) ([]byte, error)

ConvertJSONLStreamWithOptions converts JSONL to TOON with streaming mode and custom options.

func ConvertJSONLString 

func ConvertJSONLString(jsonlStr string) (string, error)

ConvertJSONLString converts a JSONL string to TOON string.

func ConvertJSONLWithOptions 

func ConvertJSONLWithOptions(jsonlBytes []byte, opts ...ConverterOption) ([]byte, error)

ConvertJSONLWithOptions converts JSONL to TOON with custom options.

func ConvertString 

func ConvertString(jsonStr string) (string, error)

ConvertString converts a JSON string to TOON string.

func ConvertToJSON 

func ConvertToJSON(toonBytes []byte) ([]byte, error)

ConvertToJSON converts TOON bytes to JSON bytes.

func ConvertToJSONFile 

func ConvertToJSONFile(path string) ([]byte, error)

ConvertToJSONFile converts a TOON file to JSON bytes.

func ConvertToJSONFileString 

func ConvertToJSONFileString(path string) (string, error)

ConvertToJSONFileString converts a TOON file to JSON string.

func ConvertToJSONFromReader 

func ConvertToJSONFromReader(r io.Reader) ([]byte, error)

ConvertToJSONFromReader converts TOON from an io.Reader to JSON bytes.

func ConvertToJSONString 

func ConvertToJSONString(toonStr string) (string, error)

ConvertToJSONString converts TOON string to JSON string.

func ConvertToJSONWithOptions 

func ConvertToJSONWithOptions(toonBytes []byte, opts ...DecodeOption) ([]byte, error)

ConvertToJSONWithOptions converts TOON to JSON with custom options.

func ConvertWithOptions 

func ConvertWithOptions(jsonBytes []byte, opts ...ConverterOption) ([]byte, error)

ConvertWithOptions converts JSON bytes to TOON with custom options.

func DecodeFromReader 

func DecodeFromReader(r io.Reader) ([]byte, error)

DecodeFromReader reads TOON from reader and returns JSON bytes.

func DecodeToJSON 

func DecodeToJSON(toon []byte) ([]byte, error)

DecodeToJSON reads TOON and returns JSON bytes.

func DecodeToJSONString 

func DecodeToJSONString(toon string) (string, error)

DecodeToJSONString reads TOON and returns JSON string.

func DecodeToJSONWithOptions 

func DecodeToJSONWithOptions(toon []byte, opts DecodeOptions) ([]byte, error)

DecodeToJSONWithOptions converts TOON to JSON with custom options.

func DetectFormat 

func DetectFormat(data []byte) string

DetectFormat detects the input format: JSON, JSONC, or JSONL.

func StripComments 

func StripComments(jsonc []byte) ([]byte, error)

StripComments removes JSONC-style comments from JSONC content. Supports:

  • Line comments: // comment to end of line
  • Block comments: /* multi-line comment */
  • Comments inside strings (not stripped)

Types

type Converter 

type Converter struct {
	// contains filtered or unexported fields
}

Converter provides streaming JSON/JSONC/JSONL to TOON conversion. It writes directly to an io.Writer without buffering the entire output.

func NewConverter 

func NewConverter(w io.Writer) *Converter

NewConverter creates a new Converter that writes TOON to w.

func NewConverterWithOptions 

func NewConverterWithOptions(w io.Writer, opts ConverterOptions) *Converter

NewConverterWithOptions creates a new Converter with custom options.

func (*Converter) Close 

func (c *Converter) Close() error

Close flushes any buffered data and releases resources.

func (*Converter) ConvertAuto 

func (c *Converter) ConvertAuto(data []byte) error

ConvertAuto auto-detects and converts the input format (JSON/JSONC/JSONL) to TOON. It examines the input to determine the format:

  • JSONC: contains // or /* */ comments
  • JSONL: multiple lines of valid JSON objects
  • JSON: standard JSON format (default)

func (*Converter) ConvertJSON 

func (c *Converter) ConvertJSON(jsonBytes []byte) error

ConvertJSON converts JSON bytes to TOON using streaming token-based parsing. It is memory-efficient for large JSON documents as it doesn't load the entire input into memory.

func (*Converter) ConvertJSONC 

func (c *Converter) ConvertJSONC(jsoncBytes []byte) error

ConvertJSONC converts JSONC (JSON with comments) to TOON. It first strips comments (// and /* */) and then converts the result.

func (*Converter) ConvertJSONL 

func (c *Converter) ConvertJSONL(jsonlBytes []byte) error

ConvertJSONL converts JSON Lines (JSONL) to TOON. Each line is parsed as a separate JSON value. If all lines are objects with the same keys, output is in tabular format.

func (*Converter) ConvertJSONLFromReader 

func (c *Converter) ConvertJSONLFromReader(r io.Reader) error

ConvertJSONLFromReader converts JSON Lines from an io.Reader to TOON. This is more memory-efficient than ConvertJSONL for large inputs.

func (*Converter) ConvertJSONLStream 

func (c *Converter) ConvertJSONLStream(r io.Reader) error

ConvertJSONLStream converts JSON Lines to TOON in streaming mode. Each line is processed and written immediately, separated by "---". This is the most memory-efficient option for large JSONL files.

type ConverterOption 

type ConverterOption func(*ConverterOptions)

ConverterOption configures the converter behavior.

func WithDelimiter 

func WithDelimiter(r rune) ConverterOption

WithDelimiter sets the table delimiter character.

func WithIndent 

func WithIndent(n int) ConverterOption

WithIndent sets the indentation level.

func WithKeyFolding 

func WithKeyFolding(mode string) ConverterOption

WithKeyFolding sets the key folding mode.

func WithStrict 

func WithStrict(strict bool) ConverterOption

WithStrict enables or disables strict mode.

type ConverterOptions 

type ConverterOptions struct {
	// Encoder configures the TOON encoder.
	Encoder EncoderOptions

	// Strict enables strict TOON spec validation.
	// When true, enforces strict compliance with TOON specification.
	// Defaults to true.
	Strict bool
}

ConverterOptions configures the JSON to TOON converter.

func DefaultConverterOptions 

func DefaultConverterOptions() ConverterOptions

DefaultConverterOptions returns the default converter options.

type DecodeOption 

type DecodeOption func(*DecodeOptions)

DecodeOption configures the decoder behavior.

func WithDecodeStrict 

func WithDecodeStrict(strict bool) DecodeOption

WithDecodeStrict enables or disables strict validation when decoding.

func WithExpandPaths 

func WithExpandPaths(expand bool) DecodeOption

WithExpandPaths enables path expansion for reconstructing nested structures from folded keys.

type DecodeOptions 

type DecodeOptions struct {
	Strict      bool
	ExpandPaths bool
}

DecodeOptions holds decoder configuration.

func DefaultDecodeOptions 

func DefaultDecodeOptions() DecodeOptions

DefaultDecodeOptions returns the default decoder options.

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads TOON format and converts to JSON.

func NewDecoder 

func NewDecoder(r io.Reader) *Decoder

NewDecoder creates a new Decoder reading from r. The decoder will use a default indent of 2 spaces and strict validation.

func NewDecoderWithOptions 

func NewDecoderWithOptions(r io.Reader, opts DecodeOptions) *Decoder

NewDecoderWithOptions creates a new Decoder with custom options. Use DefaultDecodeOptions() as a starting point and modify as needed.

func (*Decoder) Decode 

func (d *Decoder) Decode() (interface{}, error)

Decode reads TOON from the reader and returns a JSON-compatible value. The returned value will be one of:

  • map[string]interface{} for objects
  • []interface{} for arrays
  • string, float64, bool, or nil for primitives

Returns (nil, nil) if the input is empty.

On error, the returned error will be a *DecoderError with context information including line numbers when available.

type DecoderError 

type DecoderError struct {
	Type    string // Error type (e.g., "InvalidSyntax", "CountMismatch")
	Message string // Human-readable error message
	Line    int    // Line number (1-indexed, 0 if unknown)
	Column  int    // Column position (1-indexed, 0 if unknown)
	Context string // Input snippet for context
}

DecoderError represents a decoding error with context information.

func NewDecoderError 

func NewDecoderError(errType, message string) *DecoderError

NewDecoderError creates a new DecoderError with context.

func NewDecoderErrorWithContext 

func NewDecoderErrorWithContext(errType, message string, line, column int, context string) *DecoderError

NewDecoderErrorWithContext creates a DecoderError with line/column context.

func (*DecoderError) Error 

func (e *DecoderError) Error() string

Error implements the error interface.

func (*DecoderError) Unwrap 

func (e *DecoderError) Unwrap() error

Unwrap returns the underlying error (for error wrapping compatibility).

type DecoderOptions 

type DecoderOptions struct {
	Strict      bool
	ExpandPaths bool
}

DecoderOptions holds decoder configuration.

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes TOON format to an io.Writer.

func NewEncoder 

func NewEncoder(w io.Writer) *Encoder

NewEncoder creates a new Encoder writing to w.

func NewEncoderWithOptions 

func NewEncoderWithOptions(w io.Writer, opts EncoderOptions) *Encoder

NewEncoderWithOptions creates a new Encoder with custom options.

func (*Encoder) Encode 

func (e *Encoder) Encode(v interface{}) error

Encode encodes a JSON-compatible value to TOON.

func (*Encoder) Flush 

func (e *Encoder) Flush() error

Flush flushes any buffered data.

type EncoderOptions 

type EncoderOptions struct {
	// Indent is the number of spaces per indentation level.
	// Defaults to 2.
	Indent int

	// Delimiter is the character used to separate values in tabular arrays.
	// Options: ',', '\t', '|'
	// Defaults to ','.
	Delimiter rune

	// KeyFolding enables key folding: "off" or "safe".
	// "safe" collapses nested single-key objects into dotted paths.
	// Defaults to "off".
	KeyFolding string

	// FlattenDepth limits key folding depth. 0 means unlimited.
	// Defaults to 0 (unlimited).
	FlattenDepth int
}

EncoderOptions configures the TOON encoder behavior.

func DefaultEncoderOptions 

func DefaultEncoderOptions() EncoderOptions

DefaultEncoderOptions returns the default encoder options.

Source Files

View all Source files

Directories

Path Synopsis
cmd
j2t command
j2t converts JSON/JSONC/JSONL to TOON format (and vice versa with -r or --decode).
 j2t converts JSON/JSONC/JSONL to TOON format (and vice versa with -r or --decode).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.