jsoncompress

package module
v0.0.0-...-1c87c9b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2026 License: MIT Imports: 15 Imported by: 0

README

json-compress

Lossless JSON compression that is still JSON. No dependencies, no binary framing, nothing but the standard library. Typically 45–75% smaller, and an exact round trip.

Go reference Go version licence

go get github.com/eharain/JSON-Compress/go

import jsoncompress "github.com/eharain/JSON-Compress/go"

type User struct {
    ID      int    `json:"id"`
    Name    string `json:"name"`
    Role    string `json:"role"`
    Country string `json:"country"`
}

users := []User{
    {1, "Ada", "admin", "United Kingdom"},
    {2, "Bob", "admin", "United Kingdom"},
    {3, "Cy", "user", "United Kingdom"},
}

small, err := jsoncompress.Marshal(users)
// {"jc": 1,
//  "k": ["id", "name", "role", "country"],
//  "d": {"~t": [0, 1, 2, 3],
//        "~e": [0, 0, 0, 2],
//        "~cv": ["United Kingdom"],
//        "~r": [[1, "Ada", "admin"], [2, "Bob", "admin"], [3, "Cy", "user"]]}}
// 'country' never changes, so it is stored once. Three rows is too few for the
// string catalogue to pay off on 'role' as well - at a thousand rows it would.

back, err := jsoncompress.Unmarshal(small)   // exactly the records you started with

This is the Go implementation of the json-compress format. A document written here reads correctly in the JavaScript, .NET, Python and PHP implementations, and the other way round — which is not a claim, but a test run against a corpus the JavaScript encoder produced.

Why

JSON spends most of its bytes saying the same things over and over. A thousand records is a thousand copies of every key name; a status column with three possible values is a thousand copies of three short strings; a currency field that is always "GBP" is a thousand copies of one.

This writes each of those down once.
Key catalogue Every key name is written once and referenced by a one- or two-character token.
Columnar tables A slice of like-shaped objects becomes a column list and a rows matrix — the keys leave the rows entirely.
Constant columns A column whose value never changes is lifted out of the rows and stored once.
Dictionary columns A column drawn from a small set of strings becomes integer indices.
String catalogue Any string value that repeats often enough to pay for itself is written once.

The output is still ordinary JSON, so it goes anywhere JSON goes: an HTTP body, a WebSocket frame, a Postgres jsonb column, a queue message, a log line. Nothing in your stack needs to learn a new format.

What it measures

From go run ./examples/benchmark, which round-trips every case before reporting it:

Scenario Minified Compressed Saved + gzip Saved
API record set (1,000 users) 206.2 KB 54.8 KB 73% 13.3 KB 21%
Nested API response (orders with lines) 96.5 KB 39.9 KB 59% 9.2 KB 6%
Time series (2,000 readings) 166.7 KB 44.3 KB 73% 11.4 KB 19%
Structured logs (1,500 lines) 237.3 KB 57.3 KB 76% 20.3 KB 14%
GeoJSON feature collection (400 points) 58.9 KB 31.6 KB 46% 6.7 KB 5%
Config document (deeply nested, few repeats) 3.1 KB 2.2 KB 29% 0.5 KB −21%

The last two columns are the honest ones. Most JSON meets gzip on the way out, and gzip already removes a lot of this redundancy — so the question is what is left. On record-shaped data, a further 5–20% on top of gzip. On a small config file with little repetition, the envelope costs more than it saves, and the table says so.

The first four columns match the JavaScript, .NET, Python and PHP benchmarks figure for figure, because all five build the same documents from the same generator and all five encoders write the same bytes. The gzip columns differ by a percentage point either way: compress/flate and zlib make slightly different choices, and nothing in this library can change that.

Use it for record sets, logs, time series and API responses. For a small document you already gzip, it will not earn its keep — measure with Measure before you commit.

Where it helps most

  • API responses. Less to serialise, less to send, less to parse.
  • Cache entries. Redis and Memcached charge by the byte.
  • WebSocket traffic, where per-message compression is often off.
  • Queue and job payloads with per-message size limits.
  • JSON columns in a database. Postgres jsonb, MySQL JSON, SQLite TEXT: smaller rows, smaller indexes, smaller backups.
  • Logs and telemetry, which are almost entirely repeated keys.

Install

go get github.com/eharain/JSON-Compress/go

Go 1.21 and later. No dependencies — the standard library is all it needs.

The command line installs on its own:

go install github.com/eharain/JSON-Compress/go/cmd/json-compress@latest
json-compress stats orders.json

The value model

A restored document is built from the types JSON has and no others: nil, bool, float64, string, []any and *Object.

restored, err := jsoncompress.Unmarshal(envelope)

Object is an ordered map, and it has to be. A Go map has no order, and encoding/json writes one out in sorted order, so decoding into map[string]any would shuffle every document that passes through — and this format promises to give key order back exactly as it arrived.

row := jsoncompress.NewObject().Set("id", 1.0).Set("name", "Ada")

row.Len()             // 2
row.Keys()            // []string{"id", "name"}
key, value := row.At(0)
value, present := row.Get("name")   // present says whether the key is there at
                                    // all, which nil alone cannot

Object implements json.Marshaler and json.Unmarshaler, so it drops into code that already uses encoding/json — though json.Marshal will escape <, > and & the way Go always does. jsoncompress.FormatJSON writes the bytes this format is defined in terms of.

Input is more forgiving. Anything encoding/json can marshal can be compressed, and is read the same way it would be marshalled — struct tags and all:

jsoncompress.Marshal(rows)   // []MyStruct, []map[string]any, []*Object, ...

A map is written in sorted key order, because that is the only order a Go map has. Reach for *Object when the order matters to you.

API

Everything is in the one package.

Compress(value any, options ...Options) (*Object, error)

Compress a document into an envelope. The result is an ordinary JSON-shaped value, so it can be stored, posted or logged anywhere plain JSON goes. The input is left untouched.

Decompress(envelope any) (any, error)

Restore it. Returns a *DecodeError if the envelope is not readable, rather than something nearly right. The envelope may be an *Object or a map[string]any that encoding/json produced.

Marshal(value) / MarshalIndent(value, indent) and Unmarshal(data)

The same pair at the byte level, and the pair to prefer: because Unmarshal parses the text itself, ordinary object nodes keep the order they were written in as well.

data, err := jsoncompress.Marshal(orders)   // compress, then serialise
back, err := jsoncompress.Unmarshal(data)   // parse, then restore
DecompressIfNeeded(value) and IsCompressed(value)

Restores a compressed value, passes anything else straight through. Useful at a boundary you do not fully control — an endpoint mid-migration, a cache holding entries written by two versions of the same service.

Measure(value, options ...Options) and MeasureText(text, options ...Options)

What compression would buy, before you commit to it — here on the 1,000-user record set the benchmark builds:

report, err := jsoncompress.Measure(users)
report.Minified        // 211180
report.Compressed      // 56136
report.Percent         // 73.4
report.GzipMinified    // 17110
report.GzipCompressed  // 13596

Ratio and Percent are measured against minified JSON, not against whatever whitespace the input arrived with. MeasureText takes the document as it really came in, so Original reflects the bytes on the wire.

NewCodec(options ...Options) (*Codec, error)

Bind options once — the natural unit for a transport layer or a storage adapter. It holds nothing but its options and never changes them, so one instance built at start-up is safe to share across goroutines:

options := jsoncompress.DefaultOptions()
options.MinRows = 10

payloads, err := jsoncompress.NewCodec(options)   // once, at start-up

data, err := payloads.Marshal(payload)            // and everywhere after
back, err := payloads.Unmarshal(data)
Validate(text) and Locate(text)

Check a JSON document and say exactly where it breaks — line, column, the line itself, and what is wrong in plain words:

found := jsoncompress.Locate("{\n  \"a\": 1,\n  \"b\": oops\n}")
// Valid:    false
// Error:    expected a value, found "o"
// Position: 19,  Line: 3,  Column: 8
// Excerpt:  `  "b": oops`

encoding/json reports invalid character 'o' looking for beginning of value and an offset, which is not enough to put a cursor anywhere, so this scans the document itself against RFC 8259. It knows about single quotes, trailing commas, True/None, leading zeros, unterminated strings and bad escapes. Location has a String method, so a found problem is ready to log.

Position is a byte offset, so text[found.Position:] starts at the problem; Column counts characters, so it means what it means in an editor even when the line above holds an emoji.

Minify(text) and Beautify(text, indent)

Whitespace only, nothing else touched. Both write line feeds whatever the platform, so a document beautified on Windows matches the same document beautified anywhere else.

ParseJSON(data), FormatJSON(value) and FormatJSONIndent(value, indent)

Ordinary JSON, no envelope — but written byte for byte as JSON.stringify would write it, which encoding/json does not quite do (see Numbers below).

Normalize(value)

The conversion every entry point runs first: an arbitrary Go value read down to the model, following encoding/json for tags, embedded structs, omitempty, json.Marshaler, encoding.TextMarshaler, []byte and nil containers, and following the wire format for the rest.

ByteLength, GzipBytes, Gunzip, GzipSize

UTF-8 byte counts — what actually goes over the wire — and gzip with a zeroed timestamp, so the same input always measures the same.

Pack(records) and Unpack(packed)

The columnar idea on its own, with no envelope and no catalogue:

table, ok := jsoncompress.Pack([]any{
    jsoncompress.NewObject().Set("id", 1.0).Set("name", "Ada"),
    jsoncompress.NewObject().Set("id", 2.0).Set("name", "Bob"),
})
// {"fields": ["id", "name"], "data": [[1, "Ada"], [2, "Bob"]], "count": 2}

Useful when the other end is not Go, or when you want something a human can read straight off a network panel. PackAll / UnpackAll do the same for a slice of result sets, which is what a multi-statement query returns. Records may be *Objects, maps or structs. The second return value says whether the value was the right shape, so nothing is silently mangled.

Options

options := jsoncompress.DefaultOptions()
options.Strings = true          // de-duplicate repeated string values
options.Tables = true           // pack slices of like-shaped objects into tables
options.Constants = true        // hoist unchanging columns out of the rows
options.DictColumns = true      // encode low-cardinality string columns as indices
options.MinRows = 2             // smallest slice that may become a table
options.MaxHoleRatio = 0.3      // largest share of missing cells a table may carry
options.MinStringCount = 2      // times a string must occur before it can be catalogued
options.MinStringLength = 1     // shortest string that may be catalogued

envelope, err := jsoncompress.Compress(value, options)

Always start from DefaultOptions(). The zero Options would turn every transformation off and set a nonsense MinRows, so it is refused rather than quietly obeyed — every entry point validates, and says which field is wrong.

The two worth reaching for are MinRows, if you have many tiny slices and want them left alone, and MaxHoleRatio, if your records are sparse and you would rather pack them anyway.

What survives the round trip

Everything JSON can express, exactly:

  • Key order, including in packed tables. Where one column order cannot satisfy every row, the slice is left as objects rather than quietly reordered.
  • A missing key versus a null one. {"a": 1} and {"a": 1, "b": null} in the same slice stay different, and Object.Get tells them apart.
  • An empty object versus an empty array, which is why objects are *Object here.
  • Any string, including ones starting with a tilde, containing control characters, or made of astral-plane characters.
  • Keys that look like numbers, which most catalogue schemes reorder.
  • A key literally named __proto__, which is ordinary data here and a trap in some of the languages this format travels to.

Values JSON cannot carry are treated the way JSON treats them: non-finite numbers become null, negative zero becomes zero, and a cycle is an error rather than a hang. Everything else follows encoding/json: a json.Marshaler is asked for its JSON, an encoding.TextMarshaler for its text, a time.Time becomes RFC 3339 text, a []byte becomes base64, and a nil slice or map becomes null.

This is checked by 2,000 randomly generated documents on every test run, each compared byte for byte against what it started as, plus every combination of the options above, plus a corpus written by the JavaScript implementation, plus a go test -fuzz target for the parser.

One bound is worth knowing: a document nested more than 10,000 levels deep is refused rather than read, which is where encoding/json draws its own line.

Numbers are written the way every implementation writes them

The one place this package deliberately parts company with encoding/json is number formatting. Go writes 1e-05 where ECMAScript writes 0.00001, and 1e+21 only above a different threshold; the format follows the ECMAScript rule — positional between 1e-7 and 1e21, exponent form outside it — because that is what the wire format is defined against and what every document in the conformance corpus was written with.

The digits themselves are identical: both find the shortest decimal that reads back as the same double. Integers keep Go's exact 64-bit precision, signed and unsigned, rather than being folded through a float, so a value too large for a double survives here where it would not in JavaScript.

Strings are escaped the same way too, which means <, >, &, / and the line separators U+2028 and U+2029 are written raw where encoding/json escapes them. The document means the same thing either way; the bytes only match one of them.

Command line

go install github.com/eharain/JSON-Compress/go/cmd/json-compress@latest

json-compress stats orders.json
json-compress compress orders.json -o orders.jc.json
json-compress decompress orders.jc.json --pretty
curl -s https://api.example.com/orders | json-compress compress > orders.jc.json

stats changes nothing and reports what compression would save, with and without gzip. validate, minify and beautify are there too. Run json-compress --help for the full list.

The format

The wire format is fully specified in SPEC.md — enough to write an implementation from, in any language, without reading this one. In outline:

{
  "jc": 1,
  "k":  ["the", "key", "catalogue"],
  "s":  ["repeated", "string", "values"],
  "d":  "the encoded document"
}

Keys become short letter-first tokens; a tilde marks a catalogue reference or an escaped literal; a ~t member marks a columnar table. That is the whole of it.

Building and testing

go test ./...
go test -race ./...
go run ./examples/benchmark
go test -bench . -benchmem
go test -fuzz FuzzRoundTrip -fuzztime 60s
go test -fuzz FuzzParseAgreesWithEncodingJson -fuzztime 60s

The second fuzz target is there because the JSON reader is written against the grammar directly rather than over encoding/json's token stream, which boxes every string and number into an interface and costs an allocation for each. It is worth about five times the throughput on a record set, and the target holds it to the standard library's own answers: the same documents accepted, the same refused, the same values read out of them.

The conformance corpus lives at dotnet/tests/TechStyle.JsonCompress.Tests/fixtures/conformance.json and is shared with the .NET, Python and PHP suites rather than duplicated, so the Go tests prove this implementation agrees with the reference without Node being installed. To check the other direction as well:

node tests/interop/verify-interop.mjs    # every case through both implementations

The module lives in a subdirectory of a repository that holds five of them, so its release tags carry that subdirectory: go/v1.0.0, not v1.0.0. That is what the module proxy looks for, and go get github.com/eharain/JSON-Compress/go@v1.0.0 resolves to it.

Licence

MIT. Free for commercial use.

Built by Tech Style Ltd — issues and pull requests welcome at github.com/eharain/JSON-Compress.

Documentation

Overview

Package jsoncompress is lossless JSON compression that is still JSON.

Three ideas, applied together:

  1. Every object key is replaced by a short token and the real names are listed once, in a catalogue. A key that occurs ten thousand times is spelled out once.
  2. A slice of like-shaped objects becomes a table - one column list and a matrix of rows - so the keys disappear from the rows entirely. Columns whose value never changes are hoisted out of the rows altogether, and columns drawn from a small set of strings become integer indices.
  3. String values that repeat are moved into a second catalogue and referenced by token.

The output is an ordinary JSON value with no binary framing, so it survives anything that carries JSON: an HTTP body, a WebSocket frame, a jsonb column, a queue message, a log line. 

envelope, err := jsoncompress.Compress([]any{
	map[string]any{"id": 1, "role": "admin"},
	map[string]any{"id": 2, "role": "admin"},
})
back, err := jsoncompress.Decompress(envelope)

The wire format is specified in SPEC.md at the root of the repository, and this implementation writes the same bytes as the JavaScript, .NET, Python and PHP ones.

The value model

A restored document is built from the types JSON has and no others: nil, bool, float64, string, []any and *Object. Objects are *Object rather than map[string]any because a Go map has no order and this format promises to give key order back exactly as it arrived.

Input is more forgiving. Anything encoding/json can marshal can be compressed - structs with their tags, maps, slices, json.Marshaler, encoding.TextMarshaler, time.Time - and is read the way encoding/json reads it, with map keys sorted because a Go map cannot say what order it wants.

Example

The shape the format is built for: a record set, where every row repeats every key name and most rows repeat the values too. 

package main

import (
	"fmt"

	jsoncompress "github.com/eharain/JSON-Compress/go"
)

type user struct {
	ID      int    `json:"id"`
	Name    string `json:"name"`
	Role    string `json:"role"`
	Country string `json:"country"`
}

func main() {
	users := []user{
		{1, "Ada", "admin", "United Kingdom"},
		{2, "Bob", "admin", "United Kingdom"},
		{3, "Cy", "user", "United Kingdom"},
	}

	small, err := jsoncompress.Marshal(users)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(small))

	back, err := jsoncompress.Unmarshal(small)
	if err != nil {
		panic(err)
	}
	restored, err := jsoncompress.FormatJSON(back)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(restored))

}

Output:
{"jc":1,"k":["id","name","role","country"],"d":{"~t":[0,1,2,3],"~e":[0,0,0,2],"~cv":["United Kingdom"],"~r":[[1,"Ada","admin"],[2,"Bob","admin"],[3,"Cy","user"]]}}
[{"id":1,"name":"Ada","role":"admin","country":"United Kingdom"},{"id":2,"name":"Bob","role":"admin","country":"United Kingdom"},{"id":3,"name":"Cy","role":"user","country":"United Kingdom"}]

Index

Examples

Constants

View Source 
const FormatVersion = 1

FormatVersion is the version of the wire format this build reads and writes. It is bumped only on a breaking format change.

View Source 
const Version = "1.0.0"

Version is the version of this implementation, which is not the same thing as the version of the wire format (FormatVersion).

Variables

This section is empty.

Functions

func Beautify 

func Beautify(text string, indent int) (string, error)

Beautify re-indents a JSON document. Line feeds are written whatever the platform, so a document beautified on Windows matches the same document beautified anywhere else.

func ByteLength 

func ByteLength(text string) int

ByteLength is the length of a string in UTF-8 bytes - what goes over the wire, as opposed to the number of characters in it.

func Decompress 

func Decompress(envelope any) (any, error)

Decompress restores the original value from a json-compress envelope.

The envelope may be an *Object - from Compress, or from ParseJSON - or a map[string]any, as encoding/json would have decoded it. Note that a map has no key order to give back, so an envelope that has been through encoding/json restores its ordinary object nodes in catalogue order rather than the order they were written in; packed tables carry their own order and are unaffected. Unmarshal avoids the question by parsing the text itself.

func DecompressIfNeeded 

func DecompressIfNeeded(value any) (any, error)

DecompressIfNeeded restores a value whether or not it was compressed.

Handy at a boundary you do not fully control - an endpoint being migrated, a cache holding entries written by two versions of the same service.

func FormatJSON 

func FormatJSON(value any) ([]byte, error)

FormatJSON serialises a value as JSON text, character for character as JSON.stringify would write it.

That is not quite what encoding/json writes. Go escapes <, > and & for the benefit of HTML documents, and switches to exponent form at different magnitudes than ECMAScript does - so 0.00001 comes out as 1e-05 there and 0.00001 here. The wire format is defined against the ECMAScript spelling, and every implementation of it agrees byte for byte, which is what this preserves.

Values are normalised on the way in, so anything encoding/json can marshal can be written here.

func FormatJSONIndent 

func FormatJSONIndent(value any, indent int) ([]byte, error)

FormatJSONIndent is FormatJSON with each level indented by the given number of spaces, capped at ten as JSON.stringify caps it. An indent of zero or less writes the compact form.

func Gunzip 

func Gunzip(data []byte) ([]byte, error)

Gunzip reverses GzipBytes.

func GzipBytes 

func GzipBytes(data []byte) ([]byte, error)

GzipBytes gzips a document.

The header carries no timestamp, so the same input always gives the same bytes and a size can be compared across runs. The level is zlib's own default, which is what browsers use for CompressionStream, so a figure measured here is comparable with one measured in the JavaScript implementation.

func GzipSize 

func GzipSize(data []byte) int

GzipSize is how many bytes a document takes once gzipped. Compression cannot fail on a byte slice, so this reports the size and no error; it is the figure Report carries.

func IsCompressed 

func IsCompressed(value any) bool

IsCompressed reports whether a value looks like a json-compress envelope this build can read.

func IsPacked 

func IsPacked(value any) bool

IsPacked reports whether a value has the shape Unpack knows how to read.

func Marshal 

func Marshal(value any, options ...Options) ([]byte, error)

Marshal compresses a value and serialises it in one step. It is Compress followed by FormatJSON.

func MarshalIndent 

func MarshalIndent(value any, indent int, options ...Options) ([]byte, error)

MarshalIndent is Marshal with the envelope indented by the given number of spaces.

func Minify 

func Minify(text string) (string, error)

Minify strips the whitespace from a JSON document without changing what it means.

func Normalize 

func Normalize(value any) (any, error)

Normalize brings an arbitrary Go value down to the JSON value model: nil, bool, string, the three number types the model keeps, []any and *Object.

The rules are the ones encoding/json follows, so that a value compressed here and a value marshalled there describe the same document:

  • a json.Marshaler is asked for its JSON, so a time.Time becomes the string it always writes, and a json.RawMessage is read as the document it holds;
  • an encoding.TextMarshaler becomes its text;
  • struct fields follow their json tags, including omitempty, and embedded structs are flattened where encoding/json flattens them;
  • a []byte becomes base64 text;
  • a nil slice, map, pointer or interface becomes null;
  • map keys are sorted, because a Go map has no order of its own.

Two rules come from the wire format rather than from encoding/json, and are the ones every implementation shares: a non-finite number becomes null rather than an error, and negative zero becomes zero. A cycle is an error, not a hang.

Integers keep their exact 64-bit value rather than being folded through a double, so a number too large for a float64 survives here where it would not in JavaScript.

func PackAll 

func PackAll(resultSets any) ([]any, bool)

PackAll packs every result set in a slice of result sets.

Database drivers that return several result sets from one call hand back a slice of slices; this packs each inner one and leaves the outer one alone. A set that cannot be packed is carried through as it was, so the result holds a *PackedTable where packing worked and the original set where it did not.

func ParseJSON 

func ParseJSON(data []byte) (any, error)

ParseJSON reads JSON text into the value model: nil, bool, float64, string, []any and *Object.

It is JSON.parse, with objects that remember their key order. Numbers become float64, as they do in every other implementation of this format and in encoding/json.

func ParseJSONString 

func ParseJSONString(text string) (any, error)

ParseJSONString is ParseJSON for a string.

func Unmarshal 

func Unmarshal(data []byte) (any, error)

Unmarshal parses a compressed document and restores the original value. It is ParseJSON followed by Decompress, and because it does the parsing itself the restored document keeps the key order it was written with.

func Unpack 

func Unpack(packed any) ([]any, bool)

Unpack rebuilds the records Pack flattened, and reports whether the value was a packed table at all.

func UnpackAll 

func UnpackAll(resultSets any) ([]any, bool)

UnpackAll unpacks every result set PackAll packed, carrying through anything that was not a packed table.

Types

type Codec 

type Codec struct {
	// contains filtered or unexported fields
}

Codec is a set of options, bound once and reused.

A codec is the natural unit when the same options apply everywhere - a transport layer, a cache adapter, a storage helper - and it keeps the option struct off every call site. It holds nothing but its options, which it does not change, so one instance built at start-up is safe to share across goroutines.

Example

A codec binds a set of options once, for a transport or storage layer that uses the same ones everywhere. 

package main

import (
	"fmt"

	jsoncompress "github.com/eharain/JSON-Compress/go"
)

func main() {
	options := jsoncompress.DefaultOptions()
	options.MinRows = 10

	codec, err := jsoncompress.NewCodec(options)
	if err != nil {
		panic(err)
	}

	rows := []any{
		jsoncompress.NewObject().Set("a", 1.0),
		jsoncompress.NewObject().Set("a", 2.0),
	}

	// Two rows is under the threshold, so they stay as objects.
	data, err := codec.Marshal(rows)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(data))

}

Output:
{"jc":1,"k":["a"],"d":[{"A":1},{"A":2}]}

func NewCodec 

func NewCodec(options ...Options) (*Codec, error)

NewCodec binds a set of options, validating them once rather than on every call.

func (*Codec) Compress 

func (c *Codec) Compress(value any) (*Object, error)

Compress packs a value with this codec's options.

func (*Codec) Decompress 

func (c *Codec) Decompress(envelope any) (any, error)

Decompress restores a compressed value.

func (*Codec) Marshal 

func (c *Codec) Marshal(value any) ([]byte, error)

Marshal compresses a value with this codec's options and serialises it.

func (*Codec) Measure 

func (c *Codec) Measure(value any) (Report, error)

Measure reports what this codec's options buy on a value.

func (*Codec) Options 

func (c *Codec) Options() Options

Options returns the options this codec was built with.

func (*Codec) Unmarshal 

func (c *Codec) Unmarshal(data []byte) (any, error)

Unmarshal parses and restores a compressed document.

type DecodeError 

type DecodeError struct {
	// contains filtered or unexported fields
}

DecodeError says that a document is not a readable json-compress envelope.

func (*DecodeError) Error 

func (e *DecodeError) Error() string

type Location 

type Location struct {
	// Valid is true when the document is well-formed JSON, in which case every
	// other field is zero.
	Valid bool
	// Error says what is wrong, in plain words.
	Error string
	// Position is the byte offset of the problem, so text[Position:] starts at
	// it.
	Position int
	// Line is the line number, counted from one.
	Line int
	// Column is the column number, counted from one in characters - so a column
	// means what it means in an editor even when the line above holds an emoji.
	Column int
	// Excerpt is the line the problem is on.
	Excerpt string
}

Location is what a scan of a JSON document found.

func Locate 

func Locate(text string) Location

Locate checks a JSON document and, if it is broken, says exactly where.

It is only worth running after a parse has already failed, so the cost of a second pass falls on documents that were not going to work anyway.

Example

Locate says exactly where a document stops being JSON. 

package main

import (
	"fmt"

	jsoncompress "github.com/eharain/JSON-Compress/go"
)

func main() {
	found := jsoncompress.Locate("{\n  \"a\": 1,\n  \"b\": oops\n}")

	fmt.Println(found.Valid)
	fmt.Println(found.Error)
	fmt.Println(found.Line, found.Column)
	fmt.Println(found.Excerpt)
	fmt.Println(found)

}

Output:
false
expected a value, found "o"
3 8
  "b": oops
invalid JSON at line 3, column 8: expected a value, found "o"

func Validate 

func Validate(text string) Location

Validate checks that a string is valid JSON, and says exactly where it stops being so.

The verdict comes from the parser, so it is the same verdict ParseJSON would give. The location comes from a scan of the document, because parser messages are written for parser authors and often name the wrong culprit.

func (Location) String 

func (l Location) String() string

String is a one-line summary, ready to log.

type Object 

type Object struct {
	// contains filtered or unexported fields
}

Object is a JSON object that remembers the order its keys arrived in.

A Go map does not, and encoding/json writes one out in sorted order, so a document decoded into map[string]any comes back with its members shuffled. This format promises the opposite - the same values and the same key order - so objects are represented by this type instead.

The zero value is an empty object and is ready to use. An Object is not safe for concurrent modification.

Example

An Object keeps the order its keys arrived in, and can say whether a key is present at all - which a nil value cannot. 

package main

import (
	"fmt"

	jsoncompress "github.com/eharain/JSON-Compress/go"
)

func main() {
	row := jsoncompress.NewObject().Set("id", 1.0).Set("name", "Ada").Set("note", nil)

	fmt.Println(row.Len(), row.Keys())

	note, present := row.Get("note")
	fmt.Println(note, present)

	missing, present := row.Get("nickname")
	fmt.Println(missing, present)

	text, _ := jsoncompress.FormatJSON(row)
	fmt.Println(string(text))

}

Output:
3 [id name note]
<nil> true
<nil> false
{"id":1,"name":"Ada","note":null}

func Compress 

func Compress(value any, options ...Options) (*Object, error)

Compress packs a value into a json-compress envelope.

The result is an ordinary JSON-shaped value, so it can be stored, posted or logged anywhere plain JSON goes; Marshal does the same thing and hands back the text. The input is left untouched, and anything encoding/json can marshal can be given (Normalize describes how each kind of value is read).

At most one set of options may be given; with none, DefaultOptions apply.

func NewObject 

func NewObject() *Object

NewObject returns an empty object.

func TextToArray 

func TextToArray(record *Object, property string, shape ...TextShape) *Object

TextToArray reads a delimited-integer column back into a slice of numbers.

Some databases return a set-valued column as a bracketed, comma-separated string. This converts one such member in place, leaving an empty slice where the column was null, and returns the record so calls can be chained.

func (*Object) At 

func (o *Object) At(i int) (string, any)

At returns the member at position i, counted from zero in key order. It panics if i is out of range, like any other index.

This is the allocation-free way to walk an object: 

for i := 0; i < obj.Len(); i++ {
	key, value := obj.At(i)
	...
}

func (*Object) Clone 

func (o *Object) Clone() *Object

Clone returns a shallow copy: the same values under the same keys, in the same order, in an object that can be changed without touching this one.

func (*Object) Delete 

func (o *Object) Delete(key string) *Object

Delete removes a member. Removing a key that is not there does nothing.

func (*Object) Get 

func (o *Object) Get(key string) (any, bool)

Get returns the value stored under key, and whether the key is present at all. A key that is present with a nil value and a key that is absent are different things, here as in the format itself.

func (*Object) Has 

func (o *Object) Has(key string) bool

Has reports whether key is present.

func (*Object) Keys 

func (o *Object) Keys() []string

Keys returns the member names, in order. The result is a copy; changing it does not change the object.

func (*Object) Len 

func (o *Object) Len() int

Len is the number of members.

func (*Object) MarshalJSON 

func (o *Object) MarshalJSON() ([]byte, error)

MarshalJSON writes the object with its members in order.

Note that encoding/json re-escapes what a json.Marshaler returns, so json.Marshal(obj) escapes <, > and & as Go always does. FormatJSON writes the bytes this format is defined in terms of.

func (*Object) Set 

func (o *Object) Set(key string, value any) *Object

Set stores a value, appending the key when it is new and leaving it where it is when it is not - the same rule JSON.parse follows for a repeated key. It returns the object, so calls can be chained.

func (*Object) UnmarshalJSON 

func (o *Object) UnmarshalJSON(data []byte) error

UnmarshalJSON reads a JSON object, keeping the order its members arrived in.

func (*Object) Value 

func (o *Object) Value(key string) any

Value returns the value stored under key, or nil when there is none.

type Options 

type Options struct {
	// Strings de-duplicates repeated string values into a catalogue.
	Strings bool
	// Tables packs slices of like-shaped objects into columnar tables.
	Tables bool
	// Constants hoists columns whose value never changes out of the rows.
	Constants bool
	// DictColumns encodes low-cardinality string columns as integer indices.
	DictColumns bool
	// MinRows is the smallest slice that may become a table.
	MinRows int
	// MaxHoleRatio is the largest share of missing cells a table may carry.
	MaxHoleRatio float64
	// MinStringCount is how many times a string must occur before it can be
	// catalogued.
	MinStringCount int
	// MinStringLength is the shortest string that may be catalogued.
	MinStringLength int
}

Options is what the encoder is allowed to do.

The zero value is not a usable set of options - every switch would be off and MinRows would be nonsense - so build one from DefaultOptions and change what you need: 

options := jsoncompress.DefaultOptions()
options.MinRows = 10
envelope, err := jsoncompress.Compress(value, options)

Every default is safe on arbitrary input, and the defaults are tuned for API responses and log batches, which is where the format earns most of its keep.

func DefaultOptions 

func DefaultOptions() Options

DefaultOptions returns the options used when none are given.

func (Options) Validate 

func (o Options) Validate() error

Validate checks that the options make sense, so that a typo surfaces at the call site rather than as a mysteriously large output.

type PackedTable 

type PackedTable struct {
	// Fields are the column names, in the order the row slices use.
	Fields []string `json:"fields"`
	// Data is one slice of values per record.
	Data [][]any `json:"data"`
	// Count is the number of records, so consumers need not measure.
	Count int `json:"count"`
}

PackedTable is a record set with its keys lifted out of the rows.

func Pack 

func Pack(records any) (*PackedTable, bool)

Pack flattens a slice of records into a field list and a rows matrix, and reports whether it could.

Fields are taken from the union of every record, in first-seen order, so a record that is missing one contributes nil in that column. Anything that is not a slice of objects is refused rather than mangled, which is what the second return value is for: 

if table, ok := jsoncompress.Pack(rows); ok {
	payload = table
}

Records may be anything Normalize reads as an object - an *Object, a map, a struct - so what a database driver hands back goes straight in.

Example

Pack is the columnar idea on its own, with no envelope and no catalogue. 

package main

import (
	"fmt"

	jsoncompress "github.com/eharain/JSON-Compress/go"
)

type user struct {
	ID      int    `json:"id"`
	Name    string `json:"name"`
	Role    string `json:"role"`
	Country string `json:"country"`
}

func main() {
	table, ok := jsoncompress.Pack([]user{{1, "Ada", "admin", "UK"}, {2, "Bob", "user", "UK"}})
	if !ok {
		panic("not a record set")
	}

	text, _ := jsoncompress.FormatJSON(table)
	fmt.Println(string(text))

}

Output:
{"fields":["id","name","role","country"],"data":[[1,"Ada","admin","UK"],[2,"Bob","user","UK"]],"count":2}

type Report 

type Report struct {
	// Original is the bytes of the input as given.
	Original int
	// Minified is the bytes once whitespace is removed.
	Minified int
	// Compressed is the bytes of the compressed envelope.
	Compressed int
	// Saved is the bytes saved against the minified form.
	Saved int
	// Ratio is Compressed divided by Minified, 0 to 1.
	Ratio float64
	// Percent is the percentage saved against the minified form.
	Percent float64
	// GzipMinified is the minified form after gzip.
	GzipMinified int
	// GzipCompressed is the compressed form after gzip.
	GzipCompressed int
}

Report is what compression bought on one value.

func Measure 

func Measure(value any, options ...Options) (Report, error)

Measure reports what compression buys on a given value.

The honest comparison is against minified JSON, not against whatever whitespace the input happened to arrive with, so Ratio and Percent both use the minified size as their baseline. Gzip figures are included because that is what most payloads meet in production, and the interesting question is whether the format still helps once gzip has had its turn.

Example

Measure says what compression would buy before anything is committed to. 

package main

import (
	"fmt"

	jsoncompress "github.com/eharain/JSON-Compress/go"
)

func main() {
	rows := make([]any, 100)
	for i := range rows {
		rows[i] = jsoncompress.NewObject().
			Set("id", float64(i)).
			Set("status", "delivered").
			Set("currency", "GBP")
	}

	report, err := jsoncompress.Measure(rows)
	if err != nil {
		panic(err)
	}
	fmt.Println(report.Minified, report.Compressed, report.Saved)

}

Output:
4791 594 4197

func MeasureText 

func MeasureText(text string, options ...Options) (Report, error)

MeasureText is Measure on a document that arrived as text, so that Original reflects what really came in, whitespace and all.

type TextShape 

type TextShape struct {
	Open      int
	Close     int
	Separator string
}

TextShape describes how a delimited column is written: how many characters to trim from each end, and what separates the values.

func DefaultTextShape 

func DefaultTextShape() TextShape

DefaultTextShape is the shape TextToArray assumes when none is given, which reads a column written as "[[[1,2,3]]]".

Source Files

View all Source files

Directories

Path Synopsis
cmd
json-compress command
Command json-compress compresses, restores and measures JSON documents.
 Command json-compress compresses, restores and measures JSON documents.
examples
benchmark command
Command benchmark reports what the format is worth, on shapes that occur in real systems.
 Command benchmark reports what the format is worth, on shapes that occur in real systems.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.