protojsonx

README

protojsonx

protojsonx is a high-performance alternative to the standard Go protobuf JSON library (google.golang.org/protobuf/encoding/protojson).

It uses a dynamic table-driven parser and unsafe pointer offset arithmetic to avoid the runtime protobuf reflection overhead in hot marshal/unmarshal paths.

⚡ Performance

Benchmarks run on an Apple M1 Pro (8 cores, Go 1.26.4), comparing standard protojson, standard binary protobuf wire format (proto), and protojsonx.

Marshalling (Serialization)

Implementation	Simple (ns/op)	Simple (allocs)	Complex (ns/op)	Complex (allocs)
protojson (Standard Lib)	5,091 ns	62	6,763 ns	69
protojsonx	842 ns	1	1,160 ns	3
proto (Binary Wire)	1,365 ns	13	1,257 ns	9

Unmarshalling (Deserialization)

Implementation	Simple (ns/op)	Simple (allocs)	Complex (ns/op)	Complex (allocs)
protojson (Standard Lib)	9,209 ns	129	11,870 ns	153
protojsonx (Standard)	2,495 ns	35	3,303 ns	28
protojsonx (ZeroCopy)	2,257 ns	13	3,190 ns	17
protojsonx (ZeroCopy + Allocator)	2,137 ns	10	3,020 ns	12
proto (Binary Wire)	1,908 ns	45	1,671 ns	33

🚀 Summary

• Marshal is about 6x faster than protojson with dramatically fewer allocations.
• Unmarshal is about 3.5-4x faster than protojson, depending on configured options.
• Marshal is competitive with binary protobuf, and is faster than binary marshal for both benchmark shapes.
• Allocations drop sharply: complex unmarshal falls from 153 allocs/op with protojson to 28 allocs/op (Standard), 17 allocs/op (ZeroCopy), or 12 allocs/op (ZeroCopy + Allocator).

Install

go get github.com/sudorandom/protojsonx

Optional integration modules:

go get github.com/sudorandom/protojsonx/protojsonxconnect
go get github.com/sudorandom/protojsonx/protojsonxgrpc

Configuration

MarshalOptions

• EmitUnpopulated bool: render fields with zero/default values.
• UseProtoNames bool: use proto snake_case names instead of JSON camelCase names.

UnmarshalOptions

• DiscardUnknown bool: ignore unknown keys after validating their JSON value.
• ZeroCopy bool: alias unescaped input string bytes directly as Go strings.
• Allocator Allocator: a custom allocator to optimize allocation of nested submessage structures.

Allocator Configuration

By default, Go's reflection API allocates nested submessages individually on the Go heap via reflect.New. In high-throughput pathways, this can lead to memory fragmentation and garbage collection pressure.

protojsonx provides a built-in pointer-stable, thread-local monotonic BumpAllocator out of the box.

Using the Built-in BumpAllocator

To use the built-in allocator, instantiate it, pass it to UnmarshalOptions, and call Reset() on the allocator to reuse its underlying buffers across requests/cycles:

// Create or reuse an allocator (not thread-safe; reuse per-goroutine or via a pool)
alloc := protojsonx.NewBumpAllocator()

// Reset allocator buffers from any previous runs
alloc.Reset()

var out MyMessage
err := protojsonx.UnmarshalOptions{
	Allocator: alloc,
}.Unmarshal(data, &out)

[!NOTE] Ensure that the lifetime of the BumpAllocator matches or outlives the decoded message. Only call Reset() when you are completely finished using the decoded structure.

Implementing a Custom Allocator

If you want to plug in your own memory management strategy (such as integrating with Go's experimental arena package or CGO-based allocators), you can implement the Allocator interface:

type Allocator interface {
	New(t reflect.Type) reflect.Value
}

ZeroCopy Caveats

When ZeroCopy is enabled, decoded string fields can point directly at the input JSON byte slice.

1. The input byte slice stays live as long as any decoded string references it.
2. Mutating or reusing the input buffer can mutate decoded strings.
3. Use it only for short-lived request-scoped data where the input buffer lifetime is clear.

ConnectRPC

Use github.com/sudorandom/protojsonx/protojsonxconnect with connect.WithCodec.

package server

import (
	"net/http"

	"connectrpc.com/connect"
	"github.com/sudorandom/protojsonx"
	"github.com/sudorandom/protojsonx/protojsonxconnect"
)

func Handler() http.Handler {
	codec := &protojsonxconnect.Codec{
		UnmarshalOptions: protojsonx.UnmarshalOptions{
			DiscardUnknown: true,
			ZeroCopy:       true,
		},
	}

	path, handler := userv1connect.NewUserServiceHandler(
		&UserServiceServer{},
		connect.WithCodec(codec),
	)

	mux := http.NewServeMux()
	mux.Handle(path, handler)
	return mux
}

See protojsonxconnect/codec_test.go for a complete runnable example test.

gRPC-Go

Use github.com/sudorandom/protojsonx/protojsonxgrpc to register a json content subtype codec.

package server

import (
	"github.com/sudorandom/protojsonx"
	"github.com/sudorandom/protojsonx/protojsonxgrpc"
)

func init() {
	protojsonxgrpc.Register(
		protojsonxgrpc.WithUnmarshalOptions(protojsonx.UnmarshalOptions{
			DiscardUnknown: true,
		}),
	)
}

Then request the JSON subtype on calls that should use the codec:

err := conn.Invoke(
	ctx,
	"/user.UserService/GetUserProfile",
	req,
	resp,
	grpc.CallContentSubtype(protojsonxgrpc.Name),
)

See protojsonxgrpc/codec_test.go for a complete runnable example test.

Development

Run tests across the root and codec modules:

just test

Keep all module files up to date:

just mod-tidy

Regenerate internal protobuf fixtures used by tests and benchmarks:

just generate-protos

Run benchmarks:

just bench

License

This project is licensed under the MIT License - see the LICENSE file for details.

Documentation

Index

• func Marshal(msg proto.Message) ([]byte, error)
• func Unmarshal(data []byte, msg proto.Message) error
• type Allocator
• type BumpAllocator
  • func NewBumpAllocator() *BumpAllocator
  • func (b *BumpAllocator) New(t reflect.Type) reflect.Value
  • func (b *BumpAllocator) Reset()
• type FieldType
• type MarshalOptions
  • func (o MarshalOptions) Marshal(msg proto.Message) ([]byte, error)
• type MessageTable
  • func GetTable(msg proto.Message) *MessageTable
• type UnmarshalOptions
  • func (o UnmarshalOptions) Unmarshal(data []byte, msg proto.Message) error

Functions

func Marshal

func Marshal(msg proto.Message) ([]byte, error)

Marshal message with default options

func Unmarshal

func Unmarshal(data []byte, msg proto.Message) error

Types

type Allocator

type Allocator interface {
	New(t reflect.Type) reflect.Value
}

type BumpAllocator

type BumpAllocator struct {
	// contains filtered or unexported fields
}

BumpAllocator is a pointer-stable monotonic allocator that pre-allocates memory chunks to reduce heap allocation overhead and GC pressure when deserializing nested submessages.

A BumpAllocator instance is NOT thread-safe and must only be used by one goroutine at a time. It can be reset and reused for subsequent operations.

func NewBumpAllocator

func NewBumpAllocator() *BumpAllocator

NewBumpAllocator creates a new BumpAllocator with an initial chunk size of 4KB.

func (*BumpAllocator) New

func (b *BumpAllocator) New(t reflect.Type) reflect.Value

New allocates a new zero value of the specified type from the allocator's memory pool and returns a reflect.Value pointing to it.

func (*BumpAllocator) Reset

func (b *BumpAllocator) Reset()

Reset clears the allocator's internal offset, allowing the pre-allocated memory chunks to be reused. Existing pointers allocated from this allocator are invalidated and must no longer be accessed.

type FieldType

type FieldType int

const (
	TypeInvalid FieldType = iota
	TypeString
	TypeInt32
	TypeInt64
	TypeUint32
	TypeUint64
	TypeFloat32
	TypeFloat64
	TypeBool
	TypeBytes
	TypeEnum
	TypeMessage
	TypeTimestamp
	TypeDuration
	TypeRepeatedString
	TypeRepeatedMessage
	TypeMapStringString
)

type MarshalOptions

type MarshalOptions struct {
	EmitUnpopulated bool
	UseProtoNames   bool
}

func (MarshalOptions) Marshal

func (o MarshalOptions) Marshal(msg proto.Message) ([]byte, error)

type MessageTable

type MessageTable struct {
	// contains filtered or unexported fields
}

func GetTable

func GetTable(msg proto.Message) *MessageTable

type UnmarshalOptions

type UnmarshalOptions struct {
	DiscardUnknown bool
	ZeroCopy       bool
	Allocator      Allocator
}

func (UnmarshalOptions) Unmarshal

func (o UnmarshalOptions) Unmarshal(data []byte, msg proto.Message) error

Unmarshal decodes JSON into msg using the compiled table for msg's generated type. A successful decode consumes the entire input, including trailing whitespace, and clears omitted fields on reused target messages.