package module
v0.12.1 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Dec 27, 2022 License: Apache-2.0 Imports: 14 Imported by: 38



CI PkgGoDev GoReportCard Coverage Telegram

Rendering Suffix Support for yaml


Before we start, let's agree on the concept renderer is just a simple function that takes some input to generate some output.

input -> [ renderer ] -> output

Now, let's start from an ordinary yaml doc:

foo: bar

Everything's static, once unmarshaled, foo gets value bar.

Now what if we would like to use some environment variable value as foo's value in unix shell style?

foo: ${FOO}

You can only get ${FOO} for your foo after unmarshaling, you have to code you own logic to map ${FOO} to some system environment variable.

WAIT A MINUTE! Isn't it actually a renderer with its usage specific to foo?

Rendering suffix is here to help to reuse your renderers, it offers a way to make your yaml doc dynamic on its own, you control config resolving in yaml, not in compiled code.

How it looks?

Let's continue with the environment variable support, say we have developed a renderer env (which calls os.ExpandEnv to map env reference to its value):

foo@env: ${FOO}

Probably you have already figured out what's going on there before I would explain:

@ is like a notifier, and notifies the renderer env with value ${FOO}, then env does its own job and generates some output using os.ExpandEnv, and at last, the output is set as foo's value.

It's simple and straightforward, right?

As you may also be wondering how can foo@env be resolved as foo since they are completely different field name! That's what we are taking care of, see How it works? section for brief introduction.


  • Use for yaml marshaling/unmarshaling


  • Field specific rendering: customizable in-place yaml data rendering
    • Add a suffix starts with @, followed by some renderer name, to your yaml field name as rendering suffix (e.g. foo@<renderer-name>: bar)
  • Type hinting: keep you data as what it supposed to be
    • Add a type hint suffix ?<some-type> to your renderer (e.g. foo@<renderer-name>?[]obj: bar suggests foo should be an array of objects using result from <renderer-name> generated with input bar)
    • See list of supported type hints
  • Data merging and patching made esay: create patching spec in yaml doc
    • Add a patching suffix ! to your renderer (after the type hint if any), feed it a patch spec object
    • Built-in jq (as select field) and rfc6902 json-patch (as patch[*]) support to select partial data from the incoming data.
  • Renderer chaining: render you data with a rendering pipeline
    • Concatenate you renderers with pipes (|), get your data rendered through the pipeline (e.g. join three renderers a, b, c -> a|b|c)
  • Supports arbitraty yaml doc without type definition in your own code.
  • Everything (except maps using any key) supports are supported.
  • Extended but still vanilla yaml, your yaml doc stays valid for all standard yaml parser.

Sample YAML doc with all features mentioned above:

# patch spec `!`
foo@a!: &foo
  # rendering pipeline with type hint
  - value@file: ./value-a.yml
    select: |-
      [ .[] | sort ]
  - { op: remove, path: /0/foo }
  - op: add
    path: /0/foo
    value: "bar"
    select: '. + "-suffix-from-jq-query"'
  select: |-
    { foo: .[0].foo, bar: .[1].bar }

bar@a!: *foo

NOTE: This module provides no renderer implementation, and the only built-in renderer is a pseudo renderer with empty name that skips rendering (output is what input is) for data patching and type hinting purpose (e.g. foo@?int!: { ... patch spec ... }). You have to roll out your own renderers. If you are in a hurry and want some handy renderers, try our, it will give you env, template and file renderers.

NOTE: This library also supports custom yaml tag !rs:<renderer> (local tag) and !<renderer> (global tag) with the same feature set as @<renderer> to your fields, but we do not recommend using that syntax as it may have issues with some yaml parser, a close example (since yaml anchor and alias cannot be used with yaml tag at the same time) of the one above is:

# not supported
# foo: !rs:a! &foo
foo: !!
  value: !rs:env|http?[]obj${FOO_FILE_PATH}
  - value: !rs:file ./value-a.yml
    select: |-
      [ .[] | sort ]
  - { op: remove, path: /0/foo }
  - op: add
    path: /0/foo
    value: "bar"
    select: '. + "-suffix-from-jq-query"'
  select: |-
    { foo: .[0].foo, bar: .[1].bar }

# not supported
# bar: !rs:a! *foo


See example_test.go

NOTE: You can find more examples in arhat-dev/dukkha

Known Limitations

See known_limitation_test.go for sample code and workaround.

  • Golang built-in map with rendering suffix applied to map key are treated as is, rendering suffix won't be recognized.
    • For map[string]any, foo@foo: bar is just a map item with key foo@foo, value bar, no data to be resolved.
    • The reason for this limitation is obvious since built-in map types don't have BaseField embedded, but it can be counterintuitive when you have a map field in a struct having BaseField embedded.

How it works?

IoC (Inversion of Control) is famous for its application in DI (Dependency Injection), but what it literally says is to control the outside world at somewhere inside the world, that's the start point.

Custom yaml unmarshaling requires custom implementation of yaml.Unmarshaler, so usually you just get your structs unmarshaled by yaml.Unmarshal directly.

We implemented something called BaseField, it lives in your struct as a embedded field, all its methods are exposed to the outside world by default, and guess what, it implements yaml.Unmarshaler, so your struct implements yaml.Unmarshaler as well.

But can you control sibling fields in a struct? Not possible in golang unless with the help of outside world, that's why we need Init() function, calling Init() with your struct actually activates the BaseField in it, Init() function tells the inner BaseField what fields the parent struct have (its sibling fields), with the help of reflection.

You only have to call Init() once for the top level struct, since then the BaseField in it knows what to do with its sibling fields, it will also search for all structs with BaseField embedded when unmarshaling, call Init() for them, until the whole yaml doc is unmarshaled.

During the unmarshaling process, BaseField.UnmarshalYAML get called by yaml.Unmarhsal, it checks the input yaml field names, if a yaml field name has a suffix starting with @, then that yaml field will be treated as using rendering suffix, BaseField parses the yaml field name to know the real field name is (e.g. foo@bar's real field name is foo) and sets the rendering pipeline with the suffix, it also saves the yaml field value on its own but not setting the actual strcut field, when you call my_struct_with_BaseField.ResolveFields(), it feeds the rendering pipeline with saved field value to generate actual field value and set that as struct field value.

All in all, BaseField handles everything related to yaml unmarshaling to support rendering suffix after initial activation, so all you need to do is to embed a BaseField as the very first field in your struct where you want to support rendering suffix and activate the top level struct (with BaseField embedded, which can be some inner field) with a Init() function call.


Have a look at or start/join a discussion on github.

A bit of history

This project originates from dukkha in July 2021.

At that time, I was looking for some task runner with matrix support to ease my life with my multi-arch oci image build pipelines and all repositories in this organization, hoping I can finally forget all these repeated work on repo maintenance.

After some time, dukkha fulfilled nearly all my expectations with buildah, golang and cosign support, and I made the original package standalone as this project to promote the idea.

Please feel free to share your thoughts about it at github discussion, feedbacks are always welcome, and also email me if you would like to join my tiny idea sharing group.


Copyright 2021 The Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
See the License for the specific language governing permissions and
limitations under the License.



Package rs provides rendering suffix support for yaml




View Source
const (
	TagNameRS = "rs"
	TagName   = TagNameRS

Struct field tag names used by this package

View Source
const (
	// ErrInterfaceTypeNotHandled is expected to be throwed by Options.InterfaceTypeHandler
	// when the requested interface type was not handled
	// it is only considered as an error when the target interface type is not `any` (interface{}),
	// or in other words,
	// it is not considered an error when InterfaceTypeHandler returned this error with `interface{}` type
	// as expected typed
	ErrInterfaceTypeNotHandled = errString("interface type not handled")


This section is empty.


func InitRecursively

func InitRecursively(fv reflect.Value, opts *Options)

InitRecursively is InitRecursivelyLimitDepth with depth = -1

func InitRecursivelyLimitDepth added in v0.11.0

func InitRecursivelyLimitDepth(fv reflect.Value, depth int, opts *Options)


func InitReflectValue added in v0.10.0

func InitReflectValue(in reflect.Value, opts *Options) bool

InitReflectValue returns true when BaseField of `in` is initilized after the call

func MergeMap

func MergeMap(
	original, additional map[string]any,

	appendList bool,
	uniqueInListItems bool,
) (map[string]any, error)

func NormalizeRawData added in v0.5.1

func NormalizeRawData(rawData any) (any, error)

NormalizeRawData decodes the rawData to common go value (nil, string, integers ...) if the type of rawData is *yaml.Node

func UniqueList added in v0.1.1

func UniqueList(dt []any) []any


type AnyObject

type AnyObject struct {
	BaseField `yaml:"-" json:"-"`
	// contains filtered or unexported fields

AnyObject is a `any` equivalent with rendering suffix support

func (*AnyObject) MarshalJSON

func (o *AnyObject) MarshalJSON() ([]byte, error)

func (*AnyObject) MarshalYAML

func (o *AnyObject) MarshalYAML() (any, error)

func (*AnyObject) NormalizedValue added in v0.3.0

func (o *AnyObject) NormalizedValue() any

NormalizedValue returns underlying value of the AnyObject with primitive types that is:

* for maps: map[string]any * for slices: []any * and primitive types for scalar types

should be called after fields being resolved

func (*AnyObject) ResolveFields

func (o *AnyObject) ResolveFields(rc RenderingHandler, depth int, names ...string) error

func (*AnyObject) UnmarshalJSON added in v0.10.0

func (o *AnyObject) UnmarshalJSON(p []byte) error

func (*AnyObject) UnmarshalYAML

func (o *AnyObject) UnmarshalYAML(n *yaml.Node) error

type AnyObjectMap added in v0.3.0

type AnyObjectMap struct {
	BaseField `yaml:"-" json:"-"`

	Data map[string]*AnyObject `rs:"inline"`

AnyObjectMap is a `map[string]any` equivalent with rendering suffix support

func (*AnyObjectMap) MarshalJSON added in v0.3.0

func (aom *AnyObjectMap) MarshalJSON() ([]byte, error)

func (*AnyObjectMap) MarshalYAML added in v0.3.0

func (aom *AnyObjectMap) MarshalYAML() (any, error)

func (*AnyObjectMap) NormalizedValue added in v0.3.0

func (aom *AnyObjectMap) NormalizedValue() map[string]any

NormalizedValue returns value of the AnyObjectMap with type map[string]any

func (*AnyObjectMap) UnmarshalJSON added in v0.10.0

func (aom *AnyObjectMap) UnmarshalJSON(p []byte) error

type BaseField

type BaseField struct {
	// contains filtered or unexported fields

BaseField is a helper struct type implementing Field interface it serves as a cache storage for encoding/decoding

func (*BaseField) HasUnresolvedField

func (f *BaseField) HasUnresolvedField() bool

func (*BaseField) Inherit

func (f *BaseField) Inherit(x *BaseField) error

Inherit unresolved fields from another BaseField useful when you are merging two structs and want to resolve only once to get all unresolved fields set

after a successful function call, f wiil be able to resolve its struct fields with unresolved values from b and its own

func (*BaseField) MarshalYAML added in v0.4.0

func (f *BaseField) MarshalYAML() (any, error)

MarshalYAML implements yaml.Marshaler by making a map of all fields known to BaseField

You can opt-out with build tag `rs_noyamlmarshaler`

func (*BaseField) ResolveFields

func (f *BaseField) ResolveFields(rc RenderingHandler, depth int, names ...string) (err error)
type MyStruct struct {
	rs.BaseField `yaml:"-"`

	MyValue string `yaml:"my_value"`

// required:
// initialize you data object
// before marshaling/unmarshaling/resolving
s := rs.Init(&MyStruct{}, nil).(*MyStruct)

// unmarshal yaml data using rendering suffix
err := yaml.Unmarshal([]byte(`{ my_value@my-renderer: 123 }`), s)
if err != nil {

err = s.ResolveFields(
	// implement your own renderer
		func(renderer string, rawData any) (result []byte, err error) {
			// usually you should have rawData normalized as golang types
			// so you don't have to tend to low level *yaml.Node objects
			rawData, err = rs.NormalizeRawData(rawData)
			if err != nil {
				return nil, err

			switch vt := rawData.(type) {
			case string:
				return []byte(vt), nil
			case []byte:
				return vt, nil
				return []byte("hello"), nil
if err != nil {



func (*BaseField) UnmarshalYAML

func (f *BaseField) UnmarshalYAML(n *yaml.Node) (err error)

UnmarshalYAML handles parsing of rendering suffix and normal yaml unmarshaling

type Field

type Field interface {
	// A Field must implement yaml.Unmarshaler interface to process rendering suffix
	// in yaml key

	// ResolveFields sets struct field values with data from unmarshaled yaml
	// depth controls to what level this function call goes
	// when depth >= 1, resolve inner fields until reaching depth limit (depth == 0)
	// when depth == 0, do nothing
	// when depth < 0, resolve recursively
	// names limits which fields to be resolved, their values are derived from
	// your field tags
	// When it's not empty, resolve specified fields only, otherwise, resolve all exported fields
	// in the struct.
	ResolveFields(rc RenderingHandler, depth int, names ...string) error

Field defines methods required for rendering suffix support

these methods are implemented by BaseField, so usually you should just embed BaseField in your struct as the very first field

func Init

func Init(in Field, opts *Options) Field

Init the BaseField embedded in your struct, the BaseField must be the first field

type Foo struct {
	rs.BaseField // or *rs.BaseField

NOTE: if the arg `in` doesn't contain BaseField or the BaseField is not the first element it does nothing and will return `in` as is.

type InterfaceTypeHandleFunc

type InterfaceTypeHandleFunc func(typ reflect.Type, yamlKey string) (any, error)

InterfaceTypeHandleFunc is a helper type to wrap your function as InterfaceTypeHandler

func (InterfaceTypeHandleFunc) Create

func (f InterfaceTypeHandleFunc) Create(typ reflect.Type, yamlKey string) (any, error)

type InterfaceTypeHandler

type InterfaceTypeHandler interface {
	// Create request interface type using yaml information
	Create(typ reflect.Type, yamlKey string) (any, error)

InterfaceTypeHandler is used when setting values for any typed field

type JSONPatchSpec

type JSONPatchSpec struct {
	BaseField `yaml:"-" json:"-"`

	Operation string `yaml:"op"`

	Path string `yaml:"path"`

	Value *yaml.Node `yaml:"value,omitempty"`

	// Resolve rendering suffix in value before being applied
	// Defaults to `true`
	Resolve *bool `yaml:"resolve"`

	// Select part of the value for patching
	// this action happens before patching
	Select string `yaml:"select"`

JSONPatchSpec per rfc6902

type MergeSource

type MergeSource struct {
	BaseField `yaml:"-" json:"-"`

	// Value for the source
	Value *yaml.Node `yaml:"value,omitempty"`

	// Resolve rendering suffix in value if any before being merged
	// Defaults to `true`
	Resolve *bool `yaml:"resolve"`

	// Select some data from the source
	Select string `yaml:"select,omitempty"`

type Options added in v0.4.0

type Options struct {
	// InterfaceTypeHandler handles interface value creation for any field
	// with a interface{...} type (including []interface{...} and map[string]interface{...})
	// for example, define a interface type Foo:
	// 		type Foo interface{ Bar() string }
	// with InterfaceTypeHandler you can return values whose type impelemts Foo during unmarshaling
	// defaults to `nil`
	InterfaceTypeHandler InterfaceTypeHandler

	// DataTagNamespace used for struct field tag parsing
	// you can set it to json to use json tag
	// supported tag values are:
	// - <first tag value as data field name>
	// - inline
	// - omitempty
	// unsupported tag values are ignored
	// defaults to `yaml`
	DataTagNamespace string

	// AllowUnknownFields whether restrict unmarshaling to known fields
	// NOTE: if there is a map field in struct with tag `rs:"inline"`
	// even when AllowUnknownFields is set to false, it still gets these unknown
	// fields marshaled to that map field
	// defaults to `false`
	AllowUnknownFields bool

	// AllowedRenderers limit renderers can be applied in rendering suffix
	// when this option is not set (nil), not renderer will be rejected
	// when set, only renderers with exact name matching will be allowed,
	// thus you may need to set an empty entry to allow pseudo built-in
	// empty renderer
	AllowedRenderers map[string]struct{}

type PatchSpec

type PatchSpec struct {
	BaseField `yaml:"-" json:"-"`

	// Value for the renderer
	// 	say we have a yaml list (`[bar]`) stored at
	// 		foo@http!:
	// 		  value:
	// 		  merge: { value: [foo] }
	// then the resolved value of foo will be `[bar, foo]`
	Value *yaml.Node `yaml:"value"`

	// Resolve rendering suffix in value if any before being patched
	// Defaults to `true`
	Resolve *bool `yaml:"resolve"`

	// Merge additional data into Value
	// this action happens first
	Merge []MergeSource `yaml:"merge,omitempty"`

	// Patch Value using standard rfc6902 json-patch
	// this action happens after merge
	Patch []JSONPatchSpec `yaml:"patch"`

	// Select part of the data as final result
	// this action happens after merge and patch
	// TODO: support jq variables
	Select string `yaml:"select"`

	// Unique to make sure elements in the sequence is unique
	// only effective when Value is yaml sequence
	Unique bool `yaml:"unique"`

	// MapListItemUnique to ensure items are unique in all merged lists respectively
	// lists with no merge data input are untouched
	MapListItemUnique bool `yaml:"map_list_item_unique"`

	// MapListAppend to append lists instead of replacing existing list
	MapListAppend bool `yaml:"map_list_append"`

PatchSpec is the input definition for renderers with a patching suffix

func (*PatchSpec) Apply added in v0.8.0

func (s *PatchSpec) Apply(rc RenderingHandler) (_ any, err error)

Apply Merge and Patch to Value, Unique is ensured if set to true

type RenderingHandleFunc added in v0.4.0

type RenderingHandleFunc func(renderer string, rawData any) (result []byte, err error)

RenderingHandleFunc is a helper type to wrap your function as RenderingHandler

func (RenderingHandleFunc) RenderYaml added in v0.4.0

func (f RenderingHandleFunc) RenderYaml(renderer string, rawData any) (result []byte, err error)

type RenderingHandler

type RenderingHandler interface {
	// RenderYaml transforms rawData to result through the renderer
	// renderer is the name of your renderer without type hint (`?<hint>`) and patch suffix (`!`)
	// rawData is the input to your renderer, which can have one of following types
	// - golang primitive types (e.g. int, float32)
	// - map[string]any
	// - []any
	// - *yaml.Node
	// when it's not *yaml.Node type, it was patched by built-in data patching support
	// (as indicated by the `!` suffix to your renderer)
	// Your renderer is responsible for casting it to its desired data type
	RenderYaml(renderer string, rawData any) (result []byte, err error)

RenderingHandler is used when resolving yaml fields

type TypeHint added in v0.3.0

type TypeHint interface {
	String() string
	// contains filtered or unexported methods

func ParseTypeHint added in v0.3.0

func ParseTypeHint(h string) (TypeHint, error)

type TypeHintBool added in v0.7.0

type TypeHintBool struct{}

func (TypeHintBool) String added in v0.7.0

func (TypeHintBool) String() string

type TypeHintFloat added in v0.3.0

type TypeHintFloat struct{}

func (TypeHintFloat) String added in v0.5.1

func (TypeHintFloat) String() string

type TypeHintInt added in v0.3.0

type TypeHintInt struct{}

func (TypeHintInt) String added in v0.5.1

func (TypeHintInt) String() string

type TypeHintNone added in v0.3.0

type TypeHintNone struct{}

func (TypeHintNone) String added in v0.5.1

func (TypeHintNone) String() string

type TypeHintObject added in v0.3.0

type TypeHintObject struct{}

func (TypeHintObject) String added in v0.5.1

func (TypeHintObject) String() string

type TypeHintObjects added in v0.3.0

type TypeHintObjects struct{}

func (TypeHintObjects) String added in v0.5.1

func (TypeHintObjects) String() string

type TypeHintStr added in v0.3.0

type TypeHintStr struct{}

func (TypeHintStr) String added in v0.5.1

func (TypeHintStr) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL