cbor

package
v0.0.0-...-9ab9da0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 22, 2024 License: MIT Imports: 13 Imported by: 0

README

Reference:

CBOR Encoding is described in RFC7049

Comparison of JSON vs CBOR

Two main areas of reduction are:

  1. CPU usage to write a log msg
  2. Size (in bytes) of log messages.

CPU Usage savings are below:

name                                    JSON time/op    CBOR time/op   delta
Info-32                                   15.3ns ± 1%    11.7ns ± 3%  -23.78%  (p=0.000 n=9+10)      
ContextFields-32                          16.2ns ± 2%    12.3ns ± 3%  -23.97%  (p=0.000 n=9+9)       
ContextAppend-32                          6.70ns ± 0%    6.20ns ± 0%   -7.44%  (p=0.000 n=9+9)       
LogFields-32                              66.4ns ± 0%    24.6ns ± 2%  -62.89%  (p=0.000 n=10+9)      
LogArrayObject-32                          911ns ±11%     768ns ± 6%  -15.64%  (p=0.000 n=10+10)     
LogFieldType/Floats-32                    70.3ns ± 2%    29.5ns ± 1%  -57.98%  (p=0.000 n=10+10)     
LogFieldType/Err-32                       14.0ns ± 3%    12.1ns ± 8%  -13.20%  (p=0.000 n=8+10)      
LogFieldType/Dur-32                       17.2ns ± 2%    13.1ns ± 1%  -24.27%  (p=0.000 n=10+9)      
LogFieldType/Object-32                    54.3ns ±11%    52.3ns ± 7%     ~     (p=0.239 n=10+10)     
LogFieldType/Ints-32                      20.3ns ± 2%    15.1ns ± 2%  -25.50%  (p=0.000 n=9+10)      
LogFieldType/Interfaces-32                 642ns ±11%     621ns ± 9%     ~     (p=0.118 n=10+10)     
LogFieldType/Interface(Objects)-32         635ns ±13%     632ns ± 9%     ~     (p=0.592 n=10+10)     
LogFieldType/Times-32                      294ns ± 0%      27ns ± 1%  -90.71%  (p=0.000 n=10+9)      
LogFieldType/Durs-32                       121ns ± 0%      33ns ± 2%  -72.44%  (p=0.000 n=9+9)       
LogFieldType/Interface(Object)-32         56.6ns ± 8%    52.3ns ± 8%   -7.54%  (p=0.007 n=10+10)     
LogFieldType/Errs-32                      17.8ns ± 3%    16.1ns ± 2%   -9.71%  (p=0.000 n=10+9)      
LogFieldType/Time-32                      40.5ns ± 1%    12.7ns ± 6%  -68.66%  (p=0.000 n=8+9)       
LogFieldType/Bool-32                      12.0ns ± 5%    10.2ns ± 2%  -15.18%  (p=0.000 n=10+8)      
LogFieldType/Bools-32                     17.2ns ± 2%    12.6ns ± 4%  -26.63%  (p=0.000 n=10+10)     
LogFieldType/Int-32                       12.3ns ± 2%    11.2ns ± 4%   -9.27%  (p=0.000 n=9+10)      
LogFieldType/Float-32                     16.7ns ± 1%    12.6ns ± 2%  -24.42%  (p=0.000 n=7+9)       
LogFieldType/Str-32                       12.7ns ± 7%    11.3ns ± 7%  -10.88%  (p=0.000 n=10+9)      
LogFieldType/Strs-32                      20.3ns ± 3%    18.2ns ± 3%  -10.25%  (p=0.000 n=9+10)      
LogFieldType/Interface-32                  183ns ±12%     175ns ± 9%     ~     (p=0.078 n=10+10)

Log message size savings is greatly dependent on the number and type of fields in the log message. Assuming this log message (with an Integer, timestamp and string, in addition to level).

{"level":"error","Fault":41650,"time":"2018-04-01T15:18:19-07:00","message":"Some Message"}

Two measurements were done for the log file sizes - one without any compression, second using compress/zlib.

Results for 10,000 log messages:

Log Format Plain File Size (in KB) Compressed File Size (in KB)
JSON 920 28
CBOR 550 28

The example used to calculate the above data is available in Examples.

Documentation

Overview

Package cbor provides primitives for storing different data in the CBOR (binary) format. CBOR is defined in RFC7049.

Index

Constants

This section is empty.

Variables

View Source 
var IntegerTimeFieldFormat = time.RFC3339

IntegerTimeFieldFormat indicates the format of timestamp decoded from an integer (time in seconds).

View Source 
var JSONMarshalFunc func(v interface{}) ([]byte, error)

JSONMarshalFunc is used to marshal interface to JSON encoded byte slice. Making it package level instead of embedded in Encoder brings some extra efforts at importing, but avoids value copy when the functions of Encoder being invoked. DO REMEMBER to set this variable at importing, or you might get a nil pointer dereference panic at runtime.

View Source 
var NanoTimeFieldFormat = time.RFC3339Nano

NanoTimeFieldFormat indicates the format of timestamp decoded from a float value (time in seconds and nanoseconds).

Functions

func AppendEmbeddedCBOR 

func AppendEmbeddedCBOR(dst, s []byte) []byte

AppendEmbeddedCBOR adds a tag and embeds input CBOR as such.

func AppendEmbeddedJSON 

func AppendEmbeddedJSON(dst, s []byte) []byte

AppendEmbeddedJSON adds a tag and embeds input JSON as such.

func Cbor2JsonManyObjects 

func Cbor2JsonManyObjects(src io.Reader, dst io.Writer) (err error)

Cbor2JsonManyObjects decodes all the CBOR Objects read from src reader. It keeps on decoding until reader returns EOF (error when reading). Decoded string is written to the dst. At the end of every CBOR Object newline is written to the output stream.

Returns error (if any) that was encountered during decode. The child functions will generate a panic when error is encountered and this function will recover non-runtime Errors and return the reason as error.

func DecodeIfBinaryToBytes 

func DecodeIfBinaryToBytes(in []byte) []byte

DecodeIfBinaryToBytes checks if the input is a binary format, if so, it will decode all Objects and return the decoded string as byte array.

func DecodeIfBinaryToString 

func DecodeIfBinaryToString(in []byte) string

DecodeIfBinaryToString converts a binary formatted log msg to a JSON formatted String Log message - suitable for printing to Console/Syslog.

func DecodeObjectToStr 

func DecodeObjectToStr(in []byte) string

DecodeObjectToStr checks if the input is a binary format, if so, it will decode a single Object and return the decoded string.

Types

type Encoder 

type Encoder struct{}

func (Encoder) AppendArrayDelim 

func (Encoder) AppendArrayDelim(dst []byte) []byte

AppendArrayDelim adds markers to indicate end of a particular array element.

func (Encoder) AppendArrayEnd 

func (Encoder) AppendArrayEnd(dst []byte) []byte

AppendArrayEnd adds markers to indicate the end of an array.

func (Encoder) AppendArrayStart 

func (Encoder) AppendArrayStart(dst []byte) []byte

AppendArrayStart adds markers to indicate the start of an array.

func (Encoder) AppendBeginMarker 

func (Encoder) AppendBeginMarker(dst []byte) []byte

AppendBeginMarker inserts a map start into the dst byte array.

func (Encoder) AppendBool 

func (Encoder) AppendBool(dst []byte, val bool) []byte

AppendBool encodes and inserts a boolean value into the dst byte array.

func (Encoder) AppendBools 

func (e Encoder) AppendBools(dst []byte, vals []bool) []byte

AppendBools encodes and inserts an array of boolean values into the dst byte array.

func (Encoder) AppendBytes 

func (Encoder) AppendBytes(dst, s []byte) []byte

AppendBytes encodes and adds an array of bytes to the dst byte array.

func (Encoder) AppendDuration 

func (e Encoder) AppendDuration(dst []byte, d time.Duration, unit time.Duration, useInt bool) []byte

AppendDuration encodes and adds a duration to the dst byte array. useInt field indicates whether to store the duration as seconds (integer) or as seconds+nanoseconds (float).

func (Encoder) AppendDurations 

func (e Encoder) AppendDurations(dst []byte, vals []time.Duration, unit time.Duration, useInt bool) []byte

AppendDurations encodes and adds an array of durations to the dst byte array. useInt field indicates whether to store the duration as seconds (integer) or as seconds+nanoseconds (float).

func (Encoder) AppendEndMarker 

func (Encoder) AppendEndMarker(dst []byte) []byte

AppendEndMarker inserts a map end into the dst byte array.

func (Encoder) AppendFloat32 

func (Encoder) AppendFloat32(dst []byte, val float32) []byte

AppendFloat32 encodes and inserts a single precision float value into the dst byte array.

func (Encoder) AppendFloat64 

func (Encoder) AppendFloat64(dst []byte, val float64) []byte

AppendFloat64 encodes and inserts a double precision float value into the dst byte array.

func (Encoder) AppendFloats32 

func (e Encoder) AppendFloats32(dst []byte, vals []float32) []byte

AppendFloats32 encodes and inserts an array of single precision float value into the dst byte array.

func (Encoder) AppendFloats64 

func (e Encoder) AppendFloats64(dst []byte, vals []float64) []byte

AppendFloats64 encodes and inserts an array of double precision float values into the dst byte array.

func (Encoder) AppendHex 

func (e Encoder) AppendHex(dst []byte, val []byte) []byte

AppendHex adds a TAG and inserts a hex bytes as a string.

func (Encoder) AppendIPAddr 

func (e Encoder) AppendIPAddr(dst []byte, ip net.IP) []byte

AppendIPAddr encodes and inserts an IP Address (IPv4 or IPv6).

func (Encoder) AppendIPPrefix 

func (e Encoder) AppendIPPrefix(dst []byte, pfx net.IPNet) []byte

AppendIPPrefix encodes and inserts an IP Address Prefix (Address + Mask Length).

func (Encoder) AppendInt 

func (Encoder) AppendInt(dst []byte, val int) []byte

AppendInt encodes and inserts an integer value into the dst byte array.

func (Encoder) AppendInt16 

func (e Encoder) AppendInt16(dst []byte, val int16) []byte

AppendInt16 encodes and inserts a int16 value into the dst byte array.

func (Encoder) AppendInt32 

func (e Encoder) AppendInt32(dst []byte, val int32) []byte

AppendInt32 encodes and inserts a int32 value into the dst byte array.

func (Encoder) AppendInt64 

func (Encoder) AppendInt64(dst []byte, val int64) []byte

AppendInt64 encodes and inserts a int64 value into the dst byte array.

func (Encoder) AppendInt8 

func (e Encoder) AppendInt8(dst []byte, val int8) []byte

AppendInt8 encodes and inserts an int8 value into the dst byte array.

func (Encoder) AppendInterface 

func (e Encoder) AppendInterface(dst []byte, i interface{}) []byte

AppendInterface takes an arbitrary object and converts it to JSON and embeds it dst.

func (Encoder) AppendInts 

func (e Encoder) AppendInts(dst []byte, vals []int) []byte

AppendInts encodes and inserts an array of integer values into the dst byte array.

func (Encoder) AppendInts16 

func (e Encoder) AppendInts16(dst []byte, vals []int16) []byte

AppendInts16 encodes and inserts an array of int16 values into the dst byte array.

func (Encoder) AppendInts32 

func (e Encoder) AppendInts32(dst []byte, vals []int32) []byte

AppendInts32 encodes and inserts an array of int32 values into the dst byte array.

func (Encoder) AppendInts64 

func (e Encoder) AppendInts64(dst []byte, vals []int64) []byte

AppendInts64 encodes and inserts an array of int64 values into the dst byte array.

func (Encoder) AppendInts8 

func (e Encoder) AppendInts8(dst []byte, vals []int8) []byte

AppendInts8 encodes and inserts an array of integer values into the dst byte array.

func (Encoder) AppendKey 

func (e Encoder) AppendKey(dst []byte, key string) []byte

AppendKey adds a key (string) to the binary encoded log message

func (Encoder) AppendLineBreak 

func (Encoder) AppendLineBreak(dst []byte) []byte

AppendLineBreak is a noop that keep API compat with json encoder.

func (Encoder) AppendMACAddr 

func (e Encoder) AppendMACAddr(dst []byte, ha net.HardwareAddr) []byte

AppendMACAddr encodes and inserts a Hardware (MAC) address.

func (Encoder) AppendNil 

func (Encoder) AppendNil(dst []byte) []byte

AppendNil inserts a 'Nil' object into the dst byte array.

func (Encoder) AppendObjectData 

func (Encoder) AppendObjectData(dst []byte, o []byte) []byte

AppendObjectData takes an object in form of a byte array and appends to dst.

func (Encoder) AppendString 

func (Encoder) AppendString(dst []byte, s string) []byte

AppendString encodes and adds a string to the dst byte array.

func (Encoder) AppendStringer 

func (e Encoder) AppendStringer(dst []byte, val fmt.Stringer) []byte

AppendStringer encodes and adds the Stringer value to the dst byte array.

func (Encoder) AppendStringers 

func (e Encoder) AppendStringers(dst []byte, vals []fmt.Stringer) []byte

AppendStringers encodes and adds an array of Stringer values to the dst byte array.

func (Encoder) AppendStrings 

func (e Encoder) AppendStrings(dst []byte, vals []string) []byte

AppendStrings encodes and adds an array of strings to the dst byte array.

func (Encoder) AppendTime 

func (e Encoder) AppendTime(dst []byte, t time.Time, unused string) []byte

AppendTime encodes and adds a timestamp to the dst byte array.

func (Encoder) AppendTimes 

func (e Encoder) AppendTimes(dst []byte, vals []time.Time, unused string) []byte

AppendTimes encodes and adds an array of timestamps to the dst byte array.

func (Encoder) AppendType 

func (e Encoder) AppendType(dst []byte, i interface{}) []byte

AppendType appends the parameter type (as a string) to the input byte slice.

func (Encoder) AppendUint 

func (e Encoder) AppendUint(dst []byte, val uint) []byte

AppendUint encodes and inserts an unsigned integer value into the dst byte array.

func (Encoder) AppendUint16 

func (e Encoder) AppendUint16(dst []byte, val uint16) []byte

AppendUint16 encodes and inserts a uint16 value into the dst byte array.

func (Encoder) AppendUint32 

func (e Encoder) AppendUint32(dst []byte, val uint32) []byte

AppendUint32 encodes and inserts a uint32 value into the dst byte array.

func (Encoder) AppendUint64 

func (Encoder) AppendUint64(dst []byte, val uint64) []byte

AppendUint64 encodes and inserts a uint64 value into the dst byte array.

func (Encoder) AppendUint8 

func (e Encoder) AppendUint8(dst []byte, val uint8) []byte

AppendUint8 encodes and inserts a unsigned int8 value into the dst byte array.

func (Encoder) AppendUints 

func (e Encoder) AppendUints(dst []byte, vals []uint) []byte

AppendUints encodes and inserts an array of unsigned integer values into the dst byte array.

func (Encoder) AppendUints16 

func (e Encoder) AppendUints16(dst []byte, vals []uint16) []byte

AppendUints16 encodes and inserts an array of uint16 values into the dst byte array.

func (Encoder) AppendUints32 

func (e Encoder) AppendUints32(dst []byte, vals []uint32) []byte

AppendUints32 encodes and inserts an array of uint32 values into the dst byte array.

func (Encoder) AppendUints64 

func (e Encoder) AppendUints64(dst []byte, vals []uint64) []byte

AppendUints64 encodes and inserts an array of uint64 values into the dst byte array.

func (Encoder) AppendUints8 

func (e Encoder) AppendUints8(dst []byte, vals []uint8) []byte

AppendUints8 encodes and inserts an array of uint8 values into the dst byte array.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.