enc

package
v0.0.0-...-c41fc0e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 19, 2026 License: MIT Imports: 15 Imported by: 1

README

enc

Replacement for encoding/json, providing an intermediate layer of abstraction between the encoded data and the typed data.

Rationale

In the built in encoding/json library, everything gets converted between []byte and specific types.

This means that when customization is needed, you end up implementing json.Unmarshaler or json.RawMessage.

This likely requires multiple scans to the data, and/or writing convoluted code.

Using an intermediate layer, which maps JSON to some intermediate types, allows for a single parsing of the []byte data in. After that, using and managing the intermediate types is easier, and less computational intensive.

A positive side effect, is that we no longer use a generic json.RawMessage which is opaque, but instead we can use the generic enc.Node (which can be type-switch-ed) or a specific one like enc.Map if we want a generic object, but not a primitive or an array.

One way to look at it is that you could use any or map[string]any, and they would work similarly, but having custom names help with readability, and also provide options for enc.Pairs which keeps the order of the entries, and enc.Digit which is arbitrary precision.

type Frame struct {
  Type string `json:"type"`
  Data enc.Node `json:"data"` // generic payload, change based on Type
}

type Op struct {
  Op string `json:"op"`
  Args enc.Map `json:"args"` // pairs
}

func handle(n enc.Node) error {
  var f Frame
  err := enc.Unmarshal(c, n, &f)
  if err != nil {
    return err
  }
  switch f.Type {
    case "error":
      var emsg string
      err := enc.Unmarshal(f.Data, &emsg) // unmarshal into a string
      if err != nil {
        return err
      }
      return errors.New(emsg) // we got an error

    case "op":
      var op Op
      err := enc.Unmarshal(f.Data, &op) // unmarshal into Op
      if err != nil {
        return err
      }
      return op.Exec()
  }
}

Marshal vs Encode

To simplify the documentation, we will use marshal when transforming an object into the interstitial types, and encoding when converting the intermediate to a JSON or MsgPack []byte.

Similarly, we say decoding when parsing the JSON []byte and unmarshalling when coercing the interstitial into an object type

More details can be found in the distinct types.

Field Names

When a struct is marshalled into enc.Node, each object field gets a single canonical name.

The name is resolved as follows:

  • If name:"foo" is present, that is the canonical name.
  • Otherwise json, yaml, or msgpack may provide the name.
  • If none of those tags are present, the Go field name is used.

All explicit names must agree. These are valid:

FieldID string `json:"id"`
FieldID string `name:"id"`
FieldID string `name:"id" json:"id" yaml:"id" msgpack:"id"`

These are rejected:

FieldID string `json:"id" yaml:"field_id"`
FieldID string `name:"id" json:"field_id"`
FieldID string `json:"id" yaml:"-"`

Skipping also has to agree across formats. If one format uses "-" while another names the field, it is treated as a configuration error.

Types

enc.Node

This is the generic interface, all interstitial types implements it and can be used as a replacement for json.RawMessage.

The following struct will only unmarshal a enc.Map or enc.Pairs:

  // {
  //   "type": "my-type",
  //   "data": { … },
  //   "cmd": ["x", "y"],
  // }

  type Frame struct {
    Type string   `json:"type"` // Coerced into string
    Data enc.Node `json:"data"` // left as-is (enc.Map)
    Path []string `json:"cmd"`  // Unmarshalled further
  }

Note: if the receiving pointer is an enc.Node or any other of the other types that implements it, or contains any of them, then the data is just shallow copied instead of unmarshalling. Another way to say it is that you can use enc.Node instead of json.RawMessage if you want to delay the unmarshalling of the data, or you can use enc.Map to delegate while forcing a json object, and so forth for enc.List and the other types.

enc.Numeric interface and enc.Integer, enc.Float and enc.Digits

When decoding a JSON number, a enc.Digits is created, which preserve exactly the same data received (this means you can decide later if you want float32, or int64, or uint16 and so on.

If then further unmarshalled, then appropriate conversions will take place, based on the target type:

  • If the target is float, int or uint (with any precision), then the digits are parsed accordingly or an error is generated

  • If the target is any, then a float64 is used (to keep compatibility with encoding/json)

When encoding, if the input is int or uint then enc.Integer is used; for float the enc.Float is used.

enc.String
  s := enc.String("foo")
enc.Bool
  b := enc.Bool(true)
enc.List
  l := enc.List{
    enc.String("answer"),
    enc.Integer(42),
  }
enc.Map

Used for generic object types, the order of the field is not kept.

  m := enc.Map{
    "type": enc.String("my-type"),
    "data": enc.Map{},
    "count": enc.Integer(42),
  }
  for k, v := range m {
    // type, data, count...
  }
enc.Pairs

This is a special type that can Marshal itself as a JSON object, but is implemented as a list of pairs, which then guarantee the order.

To keep the usability of this library high, we opted to avoid Ordered-Maps which are clumsy to use, and instead allow you to choose between the fast enc.Map, or the ordered enc.Pairs.

Each enc.Pair only stores one field name:

enc.Pair{Name: "type", Value: enc.String("filter")}

That same canonical Name is used when encoding to JSON and MsgPack.

Custom enc.Marshaler, enc.Unmarshaler vs json.Marshaler and json.Unmarshaler

This library is compatible with json.Marshaler and json.Unmarshaler, but those interfaces requires to re-encode and re-decoded []byte.

It is hence more efficient to use the new enc.Marshaler and enc.Unmarshaler.

Here an example object:

type X struct {
	Type string
	Path []string
}

func (this X) String() string {
	s := this.Type
	sep := ":"
	for _, p := range this.Path {
		s += sep + url.QueryEscape(p)
		sep = "/"
	}
	return s
}

var xRE = regexp.MustCompile(`^([a-z]+):([a-z]+(?:\/[a-z]+)*)$`)

func (this *X) Parse(c ctx.C, s string) error {
	out := xRE.FindStringSubmatch(s)
	if len(out) == 0 {
		return ctx.NewErrorf(c, "invalid X: %q", s)
	}
	log.Warnf(c, "out: %#v", out)
	this.Type = out[1]
	this.Path = []string{}
	for _, p := range strings.Split(out[2], "/") {
		this.Path = append(this.Path, url.QueryEscape(p))
	}
	return nil
}

To make it use String() and Parse() when generating enc.Node:

var _ enc.Marshaler = X{} // make sure we can marshal structs, not just pointers
var _ enc.Unmarshaler = &X{} // only pointers can be used for unmarshalling, tho

func (this X) MarshalNode(c ctx.C) (enc.Node, error) {
	return enc.String(this.String()), nil
}

func (this *X) UnmarshalNode(c ctx.C, n enc.Node) error {
	switch n := n.(type) {
	case enc.String:
		return this.Parse(c, string(n))
	default:
		return ctx.NewErrorf(c, "expected string, got %T", n)
	}
}

As you can see, it simply returns and enc.String to marshal, and only accept it back when unmarshalling.

To keep compatibility with encoding/json you might want to implement the relative versions of those methods too.

enc.Time WIP

there is an ongoing discussion if we should ad a time-like type to simplify handling of type, and enforcing RFC3339

ctx.C

One of the many reasons to implement and/or use this library is the integration with context.Context (or ctx.C)

Why this is so important can be debatable, but I often used Context as a way to inject specific logic into a generic framework.

It could be configuration at the top level, or request specific settings.

If those settings are needed in any of the UnmarshalNode or MarshalNode, to protect some data or create metrics, or automate subscriptions, you can now use a pattern like the following:

type ctxKey = struct{}

func SetupCallback(c ctx.C, fn func(x X) error) ctx.C {
  return ctx.WithValue(c, ctxKey{}, fn)
}

func NotifyCallback(c ctx.C, x X) error {
  fn := c.Value(c, ctxKey{})
  switch fn := fn.(type) {
  case nil:
    return nil
  case func(X) error:
    return fn(x)
  default:
    return ctx.NewErrorf(c, "unexpected %T: %v", fn, fn)
  }
}

Which can then be used on setup:

  c = SetupCallback(c, func(x X) error {
    if x.Type == blacklist {
      return ctx.NewErrorf(c, "invalid type: %q", x.Type)
    }
    return nil
  })

Where blacklist might depends on the language of the user, for example.

It can then be triggered while unmarshalling the requests:

func (this *X) UnmarshalNode(c ctx.C, n enc.Node) error {
  switch n := n.(type) {
  case enc.String():
    err := this.Parse(n)  
    if err != nil {
      return err
    }
    return NotifyCallback(c, *this) // if there is a callback, and returns an errors, unmarshalling is aborted
  default:
    return ctx.NewErrorf(c, "expected string, got %T", n)
  }
}

Which means that now any API you have will fail if they contains a type X which has a .Type which is blacklisted, and that blacklist might change per request, based on the user settings or permissions or so.

Another advantage of having access to the ctx.C is that you can properly use ctx/log and still retains tags which might contains information helpful for debugging

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func MarshalJSON 

func MarshalJSON(c ctx.C, from any) ([]byte, error)

func MarshalMsgPack 

func MarshalMsgPack(c ctx.C, from any) ([]byte, error)

func MustMarshalJSON 

func MustMarshalJSON(c ctx.C, from any) []byte

func MustMarshalMsgPack 

func MustMarshalMsgPack(c ctx.C, from any) []byte

func MustUnmarshal 

func MustUnmarshal(c ctx.C, n Node, into any)

func NewPipe 

func NewPipe(buf int) (Pipe, Pipe)

func Register 

func Register[T any](h *Handler, f func(ctx.C, Node) (T, error))

Register registers a factory function for type T. When h is nil, registers globally (must be called during init()). When h is non-nil, registers on that specific Handler instance (can be called anytime). Note: The factory is registered as *T, not T, so unmarshal lookup checks both T and *T.

func Unmarshal 

func Unmarshal(c ctx.C, n Node, into any) error

func UnmarshalInto 

func UnmarshalInto(c ctx.C, n Node, into any) error

skip the factory and unmarshal and directly unmarshal into the object, useful inside a custo Unmarshaler

func UnmarshalJSON 

func UnmarshalJSON(c ctx.C, j []byte, into any) error

func UnmarshalMsgPack 

func UnmarshalMsgPack(c ctx.C, data []byte, into any) error

Types

type Bool 

type Bool bool

func (Bool) GoString 

func (this Bool) GoString() string

func (Bool) String 

func (this Bool) String() string

type Bytes 

type Bytes []byte

func (Bytes) GoString 

func (this Bytes) GoString() string

func (Bytes) MarshalJSON 

func (this Bytes) MarshalJSON() ([]byte, error)

func (Bytes) String 

func (this Bytes) String() string

type Closer 

type Closer interface {
	Close(ctx.C) error
}

type Codec 

type Codec interface {
	Encode(c ctx.C, n Node) []byte
	Decode(c ctx.C, data []byte) (Node, error)
}

type Digits 

type Digits string

func (Digits) Duration 

func (this Digits) Duration(unit time.Duration) Duration

func (Digits) Float64 

func (this Digits) Float64() (float64, error)

func (Digits) GoString 

func (this Digits) GoString() string

func (Digits) Int64 

func (this Digits) Int64() (int64, error)

func (Digits) IsFloat 

func (this Digits) IsFloat() bool

func (Digits) MarshalJSON 

func (this Digits) MarshalJSON() ([]byte, error)

func (Digits) MustFloat 

func (this Digits) MustFloat() Float

func (Digits) MustInteger 

func (this Digits) MustInteger() Integer

func (Digits) MustUint 

func (this Digits) MustUint() uint64

func (Digits) String 

func (this Digits) String() string

func (Digits) Uint64 

func (this Digits) Uint64() (uint64, error)

type Duration 

type Duration time.Duration

Use this object if you want to get `1s` from time.Second the built in json library encode time.Second as 1000000000 empty string is used for zero duration

func (Duration) GoString 

func (this Duration) GoString() string

func (Duration) MarshalJSON 

func (this Duration) MarshalJSON() ([]byte, error)

func (*Duration) Parse 

func (this *Duration) Parse(s string) error

func (Duration) String 

func (this Duration) String() string

func (*Duration) UnmarshalJSON 

func (this *Duration) UnmarshalJSON(in []byte) error

type Float 

type Float float64

func (Float) Duration 

func (this Float) Duration(unit time.Duration) Duration

func (Float) Float64 

func (this Float) Float64() (float64, error)

func (Float) GoString 

func (this Float) GoString() string

func (Float) Int64 

func (this Float) Int64() (int64, error)

func (Float) MarshalJSON 

func (this Float) MarshalJSON() ([]byte, error)

func (Float) String 

func (this Float) String() string

func (Float) Uint64 

func (this Float) Uint64() (uint64, error)

type Handler 

type Handler struct {
	Factory map[reflect.Type]func(c ctx.C, n Node) (any, error)

	// called if a field is present in the NodeTree but there is no mapping on the object it's unmarshaled into
	UnhandledFields func(c ctx.C, path []any, n Node) error

	Debugf func(c ctx.C, f string, args ...any)
	// contains filtered or unexported fields
}

generic object that do the Unmarshal()/Conflate()

func (Handler) Append 

func (this Handler) Append(p any) Handler

func (Handler) Marshal 

func (this Handler) Marshal(c ctx.C, in any) (Node, error)

func (Handler) MarshalStruct 

func (this Handler) MarshalStruct(c ctx.C, in any, pairs ...Pair) (Pairs, error)

func (Handler) Unmarshal 

func (this Handler) Unmarshal(c ctx.C, n Node, into any) error

func (Handler) UnmarshalInto 

func (h Handler) UnmarshalInto(c ctx.C, n Node, into any) error

type Integer 

type Integer int64

func (Integer) Duration 

func (this Integer) Duration(unit time.Duration) Duration

func (Integer) Float64 

func (this Integer) Float64() (float64, error)

func (Integer) GoString 

func (this Integer) GoString() string

func (Integer) Int64 

func (this Integer) Int64() (int64, error)

func (Integer) MarshalJSON 

func (this Integer) MarshalJSON() ([]byte, error)

func (Integer) String 

func (this Integer) String() string

func (Integer) Uint64 

func (this Integer) Uint64() (uint64, error)

type Iterable 

type Iterable interface {
	Pairs() Pairs
}

type JSON 

type JSON struct {
	Indent bool
}

func (JSON) Decode 

func (this JSON) Decode(c ctx.C, data []byte) (Node, error)

func (JSON) Encode 

func (this JSON) Encode(c ctx.C, n Node) []byte

type List 

type List []Node

func (List) GoString 

func (this List) GoString() string

func (List) Maps 

func (this List) Maps() []Map

Maps returns a slice of all the map nodes in this list. Pairs are promoted to Maps any other types are ignored

func (List) String 

func (this List) String() string

type Map 

type Map map[string]Node

unordered map

func AsMap 

func AsMap(c ctx.C, n Node) (Map, error)

func MustMap 

func MustMap(n Node) Map

func (Map) GetString 

func (this Map) GetString(k string) (string, bool)

func (Map) GoString 

func (this Map) GoString() string

func (Map) MarshalJSON 

func (this Map) MarshalJSON() ([]byte, error)

func (Map) Pairs 

func (this Map) Pairs() Pairs

return pairs sorted alphabetically by key

func (Map) String 

func (this Map) String() string

type Marshaler 

type Marshaler interface {
	MarshalNode(ctx.C) (Node, error)
}

obejcts which implements this can override how they are marshaled (Conflate)

type MsgPack 

type MsgPack struct{}

func (MsgPack) Decode 

func (MsgPack) Decode(c ctx.C, data []byte) (Node, error)

func (MsgPack) Encode 

func (MsgPack) Encode(c ctx.C, n Node) []byte

type Nil 

type Nil struct{}

func (Nil) GoString 

func (this Nil) GoString() string

func (Nil) MarshalJSON 

func (this Nil) MarshalJSON() ([]byte, error)

func (Nil) String 

func (this Nil) String() string

type Node 

type Node interface {
	String() string
	GoString() string
	// contains filtered or unexported methods
}

func Marshal 

func Marshal(c ctx.C, in any) (Node, error)

transform an object into a enc.Node

func MustMarshal 

func MustMarshal(c ctx.C, from any) Node

type Numeric 

type Numeric interface {
	Int64() (int64, error)
	Uint64() (uint64, error)
	Float64() (float64, error)
	String() string
	Duration(unit time.Duration) Duration
}

type Pair 

type Pair struct {
	Name string

	Value Node
}

func (Pair) String 

func (this Pair) String() string

type Pairs 

type Pairs []Pair

ordered pairs, used mostly internally when Marshalling structs, to preserve the order of the fields can be used anywhere else where the order matters

func MarshalStruct 

func MarshalStruct(c ctx.C, in any, pairs ...Pair) (Pairs, error)

transform a struct into a enc.Node, using the struct tags to determine the field names and options, allow custom pairs this can be useful for adding a "type" field to a struct: 

func (f Filter) MarshalNode(c ctx.C) (enc.Node, error) {
	return enc.MarshalStruct(c, f, enc.Pair{Name: "type", Value: enc.String("filter")})
}

func (Pairs) AsMap 

func (this Pairs) AsMap() Map

func (Pairs) Delete 

func (this Pairs) Delete(keys ...string) Pairs

func (Pairs) Find 

func (this Pairs) Find(name string) Node

func (Pairs) GoString 

func (this Pairs) GoString() string

func (Pairs) MarshalJSON 

func (this Pairs) MarshalJSON() ([]byte, error)

func (Pairs) String 

func (this Pairs) String() string

type Pipe 

type Pipe struct {
	Send chan<- Node
	Recv <-chan Node
	// contains filtered or unexported fields
}

NOTE(oha): not sure if we really need this, left around for now might either add test or remove later

func (Pipe) Close 

func (this Pipe) Close(c ctx.C) error

func (Pipe) Read 

func (this Pipe) Read(c ctx.C) (Node, error)

func (Pipe) Write 

func (this Pipe) Write(c ctx.C, n Node) (err error)

type ReadWriteCloser 

type ReadWriteCloser interface {
	Reader
	Writer
	Closer
}

type ReadWriter 

type ReadWriter interface {
	Reader
	Writer
}

type Reader 

type Reader interface {
	Read(c ctx.C) (Node, error)
}

type ReaderCloser 

type ReaderCloser interface {
	Reader
	Closer
}

type String 

type String string

func (String) AsDuration 

func (this String) AsDuration() (Duration, error)

func (String) AsTime 

func (this String) AsTime() (Time, error)

func (String) GoString 

func (this String) GoString() string

func (String) MustDuration 

func (this String) MustDuration() Duration

func (String) MustTime 

func (this String) MustTime() Time

func (String) String 

func (this String) String() string

type Tag 

type Tag struct {
	Name      string
	OmitEmpty bool
	Skip      bool
}

type Time 

type Time time.Time

func (Time) GoString 

func (this Time) GoString() string

func (Time) MarshalJSON 

func (this Time) MarshalJSON() ([]byte, error)

func (*Time) Parse 

func (this *Time) Parse(s string) error

func (Time) String 

func (this Time) String() string

func (*Time) UnmarshalJSON 

func (this *Time) UnmarshalJSON(j []byte) error

type Unmarshaler 

type Unmarshaler interface {
	UnmarshalNode(ctx.C, Node) error
}

objects which implements this can override how their data is unmarshaled (Expand)

type WriteCloser 

type WriteCloser interface {
	io.Closer
	Writer
}

type Writer 

type Writer interface {
	Write(c ctx.C, n Node) error
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.