subst

README

RFC: Fancier substitutions

Introduction

Today in Plax you can substitute parameters within strings. Example:

"I like {?LIKED}."

You can also substitute parameter values structurally. Example:

{"deliver":"?ORDER"}

In this examine, ?ORDER could be bound to value represented by:

{"tacos":3,"chips":1}

Today you can also use YAML "includes" to embed some YAML into a YAML structure. This YAML inclusion supports splicing into arrays and maps. Simple example:

deliver:
  - include: items.yaml

This RFC generalizes both all of these types of substitution. Bonus: This RFC also allows you to do some processing of values before they are serialized/used.

The goal is to remain backwards-compatible.

Testing

plaxsubst is a command-line program to test/use parameter substitution independently from other Plax functionality. Also see demo.sh, which uses plaxsubst.

String substitutions

All substitutions are based on single string that contains zero or more substitution specifications. In its most general form, a substitution specification looks like this:

{VAR|PROC|SERIALIZATION}

1. VAR is the name of parameter.
2. PROC is an optional processor that consumes the parameter value given by VAR and emits an object.
3. SERIALIZATION specifies how to render the object.

The | PROC is optional, and | SERIALIZATION essentially defaults to | json.

If the specification (the stuff in the {} and including those braces) is surrounded by double-quotes, then those quotes are ignored. This behavior allows text that includes substitutions to still be valid JSON.

Variables

A variable can be a normal parameter name, which should exist as key in the given parameters.

Alternately, if the VAR starts with @, then rest of the VAR should be a filename that can be found in an include path. The contents of this file are read, and the result is deserialized based on the filename extension (e.g., .yaml, .json, .txt). The resulting object is then processed in the same manner as an object found in parameters.

Processors

Processors might turn out to be unhelpful, but the idea was too tempting to ignore.

The specification of a processor looks like

PROCESSOR_TYPE SRC ...

In this RFC, there are two processor types processors: JavaScript and jq. The SRC is either JavaScript or an jq expression according to PROCESSOR_TYPE. When using the JavaScript processor, $ is bound to the (structured) value given by VAR.

Serializations

The SERIALIZATION specifies how to render the result:

1. text: Assuming the object is string, just use that string as is (no delimiting quotes).
2. text$: Assuming the object is an array of strings, join that array with a comma and then use that result literally (without any delimiting quotes).
3. trim: Same as text but all leading and trailing whitespace is trimmed.
4. json: Serialize as JSON.
5. json$: Serialize an array as JSON and splice in those elements without the delimiting [ and ].
6. json@: Serialize an object as JSON and splice in those key/value pairs without the delimiting { and }.
7. yaml: Serialize as YAML.
8. yaml$: Serialize an array as YAML and splice in those elements without array-delimiting syntax.
9. yaml@: Serialize an object as YAML and splice in those key/value pairs without map-delimiting syntax.

The $ indicates array splicing, and the @ indicates map splicing.

Examples

See demo.sh:

echo '{"deliver":"{?want}"}' | plaxsubst -p '?want="tacos"'
{"deliver":"tacos"}

echo 'I like {?want|text}.' | plaxsubst -p '?want="tacos"'
I like tacos.

echo '{"deliver":"{?want}"}' | plaxsubst -p '?want=["tacos","chips"]'
{"deliver":["tacos","chips"]}

echo '{"deliver":["beer","{?want|json$}"]}' | plaxsubst -p '?want=["tacos","chips"]'
{"deliver":["beer","tacos","chips"]}

echo '{"deliver":"{?want}","n":{?want | js $.length | json}}' | plaxsubst -p '?want=["tacos","chips"]'
{"deliver":["tacos","chips"],"n":2}

echo '{"deliver":"{?want | jq .[0] | json}"}' | plaxsubst -p '?want=["tacos","chips"]'
{"deliver":"tacos"}

echo 'The order: {?want|text$}.' | plaxsubst -p '?want=["tacos","chips"]'
The order: tacos,chips.

echo 'The first item: {?want|jq .[0]|text}.' | plaxsubst -p '?want=["tacos","chips"]'
The first item: tacos.

echo '{"deliver":{"chips":2,"":"{?want|json@}"}}' |
    plaxsubst -p '?want={"tacos":2,"salsa":1}' -check-json-in -check-json-out
{"deliver":{"chips":2,"salsa":1,"tacos":2}}

echo 'I want <?want|text>.' | plaxsubst -d "<>" -p '?want="tacos"'
I want tacos.

echo '{"deliver":"?want"}' | plaxsubst -bind -p '?want={"tacos":3}'
{"deliver":{"tacos":3}}

echo '{"deliver":"?want | jq .[0]"}' | plaxsubst -bind -p '?want=[{"tacos":3},{"queso":1}]'
{"deliver":{"tacos":3}}

Structured substitutions

In an "object", a string of the form ?VAR will be replaced by a binding for ?VAR. (Note the lack of braces as delimiters.) The value for this binding will in general be an object, so no serialization is required. As a generalization, a string of form ?VAR | jq ... will be replaced by the result of evaluating the jq expression with ?VAR's binding as input.

Documentation

Index

• Variables
• func JSExec(ctx *Ctx, src string, env map[string]interface{}) (interface{}, error)
• func JSON(x interface{}) string
• func JSONMarshal(x interface{}) ([]byte, error)
• func StringKeys(x interface{}) (interface{}, error)
• type Bindings
  • func NewBindings() Bindings
  • func (bs *Bindings) Bind(ctx *Ctx, x interface{}) (interface{}, error)
  • func (bs *Bindings) Copy() (*Bindings, error)
  • func (bs *Bindings) Set(value string) error
  • func (bs *Bindings) SetJSON(key, value string) error
  • func (bs *Bindings) SetValue(k string, v interface{})
  • func (bs *Bindings) String() string
  • func (bs *Bindings) UnmarshalBind(ctx *Ctx, js string) (string, error)
• type Ctx
  • func NewCtx(ctx context.Context, dirs []string) *Ctx
  • func (c *Ctx) Copy() *Ctx
• type Proc
• type Subber
  • func NewSubber(delimeters string) (*Subber, error)
  • func (b *Subber) Copy() *Subber
  • func (b *Subber) Sub(ctx *Ctx, bs Bindings, s string) (string, error)
  • func (b *Subber) WithProcs(ps ...Proc) *Subber

Variables

var (
	// DefaultDelimiters are the default opening and closing
	// deliminers for a pipe expression.
	DefaultDelimiters = "{}"

	// DefaultSerialization is the default serialization for a
	// pipe expression when the Subber doesn't think it knows
	// better.
	DefaultSerialization = "json"

	// DefaultLimit is the default limit for the number of
	// recursive substitution calls a Subber will make.
	//
	// If you hit this limit intentionally, then that's pretty
	// impressive.  However, probably not something to brag about.
	DefaultLimit = 10
)

Functions

func JSExec

func JSExec(ctx *Ctx, src string, env map[string]interface{}) (interface{}, error)

JSExec executes the javascript source with the given context and environment mappings

func JSON

func JSON(x interface{}) string

JSON attempts to serialize its input with a fallback to Go '%#v' serialization.

func JSONMarshal

func JSONMarshal(x interface{}) ([]byte, error)

JSONMarshal exists to SetEscapeHTML(false) to avoid messing with <, >, and &.

Strangely (to me), SetEscapeHTML(false) also seems to change newline treatment. See TestJSONMarshal's 'newline' test.

func StringKeys

func StringKeys(x interface{}) (interface{}, error)

StringKeys will replace map[interface{}]interface{} with map[string]interface{} when that's possible and will return an error if not.

Types

type Bindings

type Bindings map[string]interface{}

Bindings maps variables to their values, which should probably all be native Go values.

func NewBindings

func NewBindings() Bindings

func (*Bindings) Bind

func (bs *Bindings) Bind(ctx *Ctx, x interface{}) (interface{}, error)

Bind replaces all structural variables in x with their corresponding values in bs (if any).

This operation is destructive (and probably shouldn't be).

An array or map should have interface{}-typed elements or values.

An unbound variable does not result in an error. See some comments in the Subber type.

func (*Bindings) Copy

func (bs *Bindings) Copy() (*Bindings, error)

Copy bindings deeply.

This method shamelessly uses JSON serialization, which can break on certains types of values.

func (*Bindings) Set

func (bs *Bindings) Set(value string) error

Set (for flag.Var) parses KEY=VALUE, where VALUE is a JSON representation of a value. Then key set to that value.

func (*Bindings) SetJSON

func (bs *Bindings) SetJSON(key, value string) error

SetJSON parses the value as JSON and stores the result at the given key.

func (*Bindings) SetValue

func (bs *Bindings) SetValue(k string, v interface{})

SetValue sets a binding explicitly (without any deserialization).

func (*Bindings) String

func (bs *Bindings) String() string

String returns a JSON repreentation of the Bindings.

func (*Bindings) UnmarshalBind

func (bs *Bindings) UnmarshalBind(ctx *Ctx, js string) (string, error)

UnmarshalBind is a Proc for a Subber.

The given string is parsed as JSON, and bs.Bind is called on that value. The result is JSON-serialized and returned.

type Ctx

type Ctx struct {
	context.Context
	IncludeDirs []string
	Tracing     bool
}

Ctx mostly provides a list of directories a Subber will search to find files.

Instead of using a context.Context-like struct, we could have IncludeDirs as a field in a Subber. However, the current approach feels slightly more natural to use if still a little embarrassing.

func NewCtx

func NewCtx(ctx context.Context, dirs []string) *Ctx

NewCtx makes a new Ctx with (a copy of) the given IncludeDirs.

func (*Ctx) Copy

func (c *Ctx) Copy() *Ctx

Copy makes a deep copy of the Ctx.

type Proc

type Proc func(*Ctx, string) (string, error)

Proc is a "processor" that a Subber can call.

A Proc computes an entire replacement for the given string.

Classic example is deserialization a JSON string input, doing structural replacement of bindings, and then reserializing. See Bindings.UnmarshalBind, which does exactly that.

type Subber

type Subber struct {
	// Procs is a list of processors that are called during
	// (recursive) substitution processing.
	Procs []Proc

	// Limit is the maximum number of recursive Sub calls.
	//
	// Default is DefaultLimit.
	Limit int

	// DefaultSerialization is the serialization when an explicit
	// serialization isn't provided and the Subber doesn't think
	// it knows better (via scruffy heuristics).
	DefaultSerialization string
	// contains filtered or unexported fields
}

Subber performs string-oriented substitutions based on a syntax like {VAR | PROC | SERIALIZATION}.

func NewSubber

func NewSubber(delimeters string) (*Subber, error)

NewSubber makes a new Subber with the pipe expression delimiters given by the first and second runes of the given string.

Uses DefaultDelimiters by default.

Uses DefaultSerialization and DefaultLimit.

func (*Subber) Copy

func (b *Subber) Copy() *Subber

Copy makes a deep copy of a Subber.

func (*Subber) Sub

func (b *Subber) Sub(ctx *Ctx, bs Bindings, s string) (string, error)

Sub performs recursive, string-based Bindings substitutions on the given input string.

func (*Subber) WithProcs

func (b *Subber) WithProcs(ps ...Proc) *Subber

WithProcs returns a copied Subber with the given Procs added.