cbor

README

Bento CBOR Processor

A Bento processor plugin for working with CBOR (Concise Binary Object Representation) data. This processor allows you to convert between JSON and CBOR formats.

Overview

CBOR is a binary data format designed for small message size with the ability to support the seamless conversion of JSON data models. This processor is RFC 7049 and RFC 8949 compliant and provides encoding and decoding capabilities for CBOR data.

Installation

Using Pre-built Binary

Download a pre-built binary from the releases page.

Usage

The CBOR processor supports four operations:

1. to_json: Converts CBOR data into JSON format
2. from_json: Converts JSON data into CBOR format
3. pack: Like from_json, plus schema-agnostic size optimisations
4. unpack: Reverses pack (and passes plain CBOR through untouched)

Configuration Examples

Convert CBOR to JSON

pipeline:
  processors:
    - cbor:
        operator: to_json

Convert JSON to CBOR

pipeline:
  processors:
    - cbor:
        operator: from_json

Pack / unpack

pipeline:
  processors:
    - cbor:
        operator: pack    # JSON -> optimised CBOR
    - cbor:
        operator: unpack  # optimised CBOR -> structured data

Size optimisation (pack)

fxamacker/cbor exposes its size optimisations (toarray, keyasint, omitempty, omitzero) as struct tags, applied by reflecting over a typed Go struct. This processor encodes map[string]any decoded from arbitrary JSON, so there are no fields for those tags to hang on and that whole class of optimisation is structurally unreachable.

pack reproduces the same effects as transforms over the generic tree, deriving the layout from the data at encode time and carrying it inside the payload. No schema, .proto file or field list is needed, and unpack reverses everything with no configuration.

struct tag	pack equivalent
toarray	homogeneous arrays of objects become a single column header plus positional rows
keyasint	field names are written once per array instead of once per element
omitzero	omit_zero: true drops null / false / 0 / "" / empty containers

The win comes from repeated field names, which dominate a typical CBOR document made of many similar objects. In a document of 1,700 objects with 36 fields each (TestPackShrinksUniformArrays), the names are re-encoded 1,700 times and pack writes them once:

rows=1700 fields=36  plain=1532762  packed=554147  ratio=2.77x

The ratio scales with how much of your payload is field names, so it climbs with longer names, more fields per object and more elements per array. Measure your own data with TestPackFixture (below) rather than relying on this figure.

Values are untouched — PreferredUnsortedEncOptions already applies ShortestFloat16, so floats that fit are 3 bytes and there is little left to reclaim on that side.

To check the ratio on your own data without copying it anywhere:

BENTO_CBOR_FIXTURE=/path/to/record.cbor go test -run TestPackFixture -v

That test asserts an exact round-trip and prints the ratio. The file is read at run time only; no fixture data is stored in this repository.

Losslessness

The defaults are lossless: pack → unpack returns exactly the input tree (asserted against real records in the test suite). Two opt-in settings trade fidelity for size:

• omit_zero collapses absent / null / zero into one state, exactly like proto3 field presence. Consumers must read a missing key as the zero value.
• allow_sparse packs arrays whose elements have differing key sets by taking the union and writing null where a value is missing — so a key that was absent comes back as an explicit null. CBOR undefined cannot be told apart from null through a generic any, so absence cannot be preserved. With allow_sparse: false (the default), heterogeneous arrays are simply left unpacked.

Migration

unpack passes non-packed payloads through unchanged, so you can switch readers to unpack first and writers to pack afterwards without a flag day. Records written before the switch stay readable.

to_json deliberately rejects packed payloads with an error pointing at unpack, rather than emitting a confusing half-decoded document. Payloads that were never packed carry no tag and are unaffected.

Options

All are advanced fields and only affect pack / unpack:

field	default	meaning
omit_zero	false	drop null / false / 0 / "" / empty containers (lossy)
allow_sparse	false	pack heterogeneous arrays using the key union (lossy)
min_rows	8	shortest array worth packing
min_fill	0.5	minimum populated-cell ratio before packing a sparse array
doc_tag	60000	CBOR tag marking a packed document
table_tag	60001	CBOR tag marking a packed table

The tag numbers are application-private and not IANA-registered; they only need to match between writer and reader, and are configurable in case they collide with something else in your pipeline.

Full Example

This example demonstrates a complete roundtrip conversion - JSON to CBOR and back to JSON:

input:
  generate:
    count: 1
    interval: 1ms
    mapping: |
      root = {
        "message": "Hello CBOR World",
        "numbers": [1, 2, 3, 4, 5],
        "nested": {
          "boolean": true,
          "null_value": null
        },
        "m": { "c": 3, "a": 1, "b": 2,}
      }

pipeline:
  processors:
    - cbor:
        operator: from_json  # Convert JSON to CBOR
    - cbor:
        operator: to_json    # Convert CBOR back to JSON

output:
  stdout: {}

logger:
  level: info

Docker

A pre built binary is also availale as a docker image:

 docker pull ghcr.io/akhenakh/bento-cbor:latest

Build

Clone the repository and build the custom Bento binary:

git clone https://github.com/akhenakh/bento-cbor.git
cd bento-cbor
go build ./cmd/bento-cbor

You can build your own binary, just load the plugin:

package main

import (
	"context"

	"github.com/warpstreamlabs/bento/public/service"

	// Import all standard Benthos components
	_ "github.com/warpstreamlabs/bento/public/components/all"

	_ "github.com/akhenakh/bento-cbor"
)

func main() {
	service.RunCLI(context.Background())
}

Technical Details

The processor uses the fxamacker/cbor/v2 library for CBOR encoding and decoding, which provides:

• RFC 7049 and RFC 8949 compliant implementation.
• High performance encoding and decoding (updated to v2.9.2 for hardened indefinite-length data encoding).
• RFC3339 UTC time encoding with nanosecond precision natively within CBOR (TimeRFC3339NanoUTC).
• Support for various CBOR data types.

Performance Optimizations

• Lazy JSON Evaluation: The to_json operator sets the decoded CBOR payload natively as Bento's internal structured message (msg.SetStructured()). This enables lazy evaluation, skipping expensive JSON byte allocation and serialization steps entirely if the next processor handles mapped structured queries or structural modifications.
• Precision Safety: The from_json processor guarantees that numbers are not incorrectly interpreted as strings inside CBOR payloads by circumventing json.Number fallback behaviors.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License.

Documentation

Index

• Constants
• func IsPacked(v any, o PackOptions) bool
• func Pack(v any, o PackOptions) any
• func Unpack(v any, o PackOptions) (any, error)
• type CBORProcessor
  • func NewProcessor(operatorStr string, opts ...Option) (*CBORProcessor, error)
  • func (cp *CBORProcessor) Close(ctx context.Context) error
  • func (cp *CBORProcessor) Process(ctx context.Context, m *service.Message) (service.MessageBatch, error)
• type Option
  • func WithPackOptions(o PackOptions) Option
• type PackOptions
  • func DefaultPackOptions() PackOptions

Constants

const (
	DefaultDocTag   uint64 = 60000 // wraps a whole packed document
	DefaultTableTag uint64 = 60001 // wraps a single packed [cols, rows] table
	DefaultMinRows         = 8
	DefaultMinFill         = 0.5
)

Defaults for the packed layout.

The tag numbers are application-private: they are NOT registered with IANA. They only need to be stable between the writer and the reader, and both are configurable so they can be moved if they ever collide with something else in your pipeline.

Functions

func IsPacked

func IsPacked(v any, o PackOptions) bool

IsPacked reports whether a decoded document carries the packed document tag.

func Pack

func Pack(v any, o PackOptions) any

Pack applies the configured transforms and wraps the result in the document tag so that Unpack (and any other reader) can identify it.

func Unpack

func Unpack(v any, o PackOptions) (any, error)

Unpack reverses Pack. Input that was never packed is returned untouched, so the same operator can read both old and new records during a rollout.

Types

type CBORProcessor

type CBORProcessor struct {
	// contains filtered or unexported fields
}

CBORProcessor processes messages by decoding CBOR data, displaying the decoded content, and re-encoding it.

func NewProcessor

func NewProcessor(operatorStr string, opts ...Option) (*CBORProcessor, error)

func (*CBORProcessor) Close

func (cp *CBORProcessor) Close(ctx context.Context) error

Close implements the service.Processor interface.

func (*CBORProcessor) Process

func (cp *CBORProcessor) Process(ctx context.Context, m *service.Message) (service.MessageBatch, error)

Process implements the service.Processor interface.

type Option

type Option func(*CBORProcessor)

Option customises a CBORProcessor at construction time.

func WithPackOptions

func WithPackOptions(o PackOptions) Option

WithPackOptions overrides the layout options used by the pack/unpack operators. It has no effect on to_json/from_json.

type PackOptions

type PackOptions struct {
	// OmitZero drops map entries whose value is null, false, 0, "" or an empty
	// container before encoding.
	//
	// This is lossy: it collapses "absent", "null" and "zero" into a single
	// state, exactly like proto3 field presence. Consumers must treat a missing
	// key as the zero value. Off by default.
	OmitZero bool

	// AllowSparse permits packing arrays whose elements do not all share the
	// same key set, by taking the union of keys and writing null in the cells
	// that have no value.
	//
	// This is lossy: a key that was absent from a row round-trips as an explicit
	// null, because CBOR "undefined" decodes to Go nil and cannot be told apart
	// from null through a generic any. Off by default, which makes packing
	// strictly lossless at the cost of skipping heterogeneous arrays.
	AllowSparse bool

	// MinRows is the shortest array worth packing. Below this the column header
	// costs more than the repeated keys save.
	MinRows int

	// MinFill is the minimum ratio of populated cells (0..1) required before a
	// sparse array is packed. Only consulted when AllowSparse is set.
	MinFill float64

	// DocTag and TableTag are the CBOR tag numbers used to mark a packed
	// document and a packed table respectively.
	DocTag   uint64
	TableTag uint64
}

PackOptions controls the schema-agnostic size optimisations applied by the "pack" operator.

None of these require knowing the shape of the data up front: the column names are derived from the values at encode time and travel inside the payload, so "unpack" can reverse the transform with no configuration.

func DefaultPackOptions

func DefaultPackOptions() PackOptions

DefaultPackOptions returns the lossless defaults.