dynblock

README

HCL Dynamic Blocks Extension

This HCL extension implements a special block type named "dynamic" that can be used to dynamically generate blocks of other types by iterating over collection values.

Normally the block structure in an HCL configuration file is rigid, even though dynamic expressions can be used within attribute values. This is convenient for most applications since it allows the overall structure of the document to be decoded easily, but in some applications it is desirable to allow dynamic block generation within certain portions of the configuration.

Dynamic block generation is performed using the dynamic block type:

toplevel {
  nested {
    foo = "static block 1"
  }

  dynamic "nested" {
    for_each = ["a", "b", "c"]
    iterator = nested
    content {
      foo = "dynamic block ${nested.value}"
    }
  }

  nested {
    foo = "static block 2"
  }
}

The above is interpreted as if it were written as follows:

toplevel {
  nested {
    foo = "static block 1"
  }

  nested {
    foo = "dynamic block a"
  }

  nested {
    foo = "dynamic block b"
  }

  nested {
    foo = "dynamic block c"
  }

  nested {
    foo = "static block 2"
  }
}

Since HCL block syntax is not normally exposed to the possibility of unknown values, this extension must make some compromises when asked to iterate over an unknown collection. If the length of the collection cannot be statically recognized (because it is an unknown value of list, map, or set type) then the dynamic construct will generate a single dynamic block whose iterator key and value are both unknown values of the dynamic pseudo-type, thus causing any attribute values derived from iteration to appear as unknown values. There is no explicit representation of the fact that the length of the collection may eventually be different than one.

Usage

Pass a body to function Expand to obtain a new body that will, on access to its content, evaluate and expand any nested dynamic blocks. Dynamic block processing is also automatically propagated into any nested blocks that are returned, allowing users to nest dynamic blocks inside one another and to nest dynamic blocks inside other static blocks.

HCL structural decoding does not normally have access to an EvalContext, so any variables and functions that should be available to the for_each and labels expressions must be passed in when calling Expand. Expressions within the content block are evaluated separately and so can be passed a separate EvalContext if desired, during normal attribute expression evaluation.

Detecting Variables

Some applications dynamically generate an EvalContext by analyzing which variables are referenced by an expression before evaluating it.

This unfortunately requires some extra effort when this analysis is required for the context passed to Expand: the HCL API requires a schema to be provided in order to do any analysis of the blocks in a body, but the low-level schema model provides a description of only one level of nested blocks at a time, and thus a new schema must be provided for each additional level of nesting.

To make this arduous process as convenient as possible, this package provides a helper function WalkForEachVariables, which returns a WalkVariablesNode instance that can be used to find variables directly in a given body and also determine which nested blocks require recursive calls. Using this mechanism requires that the caller be able to look up a schema given a nested block type. For simple formats where a specific block type name always has the same schema regardless of context, a walk can be implemented as follows:

func walkVariables(node dynblock.WalkVariablesNode, schema *hcl.BodySchema) []hcl.Traversal {
	vars, children := node.Visit(schema)

	for _, child := range children {
		var childSchema *hcl.BodySchema
		switch child.BlockTypeName {
		case "a":
			childSchema = &hcl.BodySchema{
				Blocks: []hcl.BlockHeaderSchema{
					{
						Type:       "b",
						LabelNames: []string{"key"},
					},
				},
			}
		case "b":
			childSchema = &hcl.BodySchema{
				Attributes: []hcl.AttributeSchema{
					{
						Name:     "val",
						Required: true,
					},
				},
			}
		default:
			// Should never happen, because the above cases should be exhaustive
			// for the application's configuration format.
			panic(fmt.Errorf("can't find schema for unknown block type %q", child.BlockTypeName))
		}

		vars = append(vars, testWalkAndAccumVars(child.Node, childSchema)...)
	}
}

Detecting Variables with hcldec Specifications

For applications that use the higher-level hcldec package to decode nested configuration structures into cty values, the same specification can be used to automatically drive the recursive variable-detection walk described above.

The helper function ForEachVariablesHCLDec allows an entire recursive configuration structure to be analyzed in a single call given a hcldec.Spec that describes the nested block structure. This means a hcldec-based application can support dynamic blocks with only a little additional effort:

func decodeBody(body hcl.Body, spec hcldec.Spec) (cty.Value, hcl.Diagnostics) {
	// Determine which variables are needed to expand dynamic blocks
	neededForDynamic := dynblock.ForEachVariablesHCLDec(body, spec)

	// Build a suitable EvalContext and expand dynamic blocks
	dynCtx := buildEvalContext(neededForDynamic)
	dynBody := dynblock.Expand(body, dynCtx)

	// Determine which variables are needed to fully decode the expanded body
	// This will analyze expressions that came both from static blocks in the
	// original body and from blocks that were dynamically added by Expand.
	neededForDecode := hcldec.Variables(dynBody, spec)

	// Build a suitable EvalContext and then fully decode the body as per the
	// hcldec specification.
	decCtx := buildEvalContext(neededForDecode)
	return hcldec.Decode(dynBody, spec, decCtx)
}

func buildEvalContext(needed []hcl.Traversal) *hcl.EvalContext {
	// (to be implemented by your application)
}

Performance

This extension is going quite harshly against the grain of the HCL API, and so it uses lots of wrapping objects and temporary data structures to get its work done. HCL in general is not suitable for use in high-performance situations or situations sensitive to memory pressure, but that is especially true for this extension.

Documentation

Index

• func Expand(body hcl.Body, ctx *hcl.EvalContext) hcl.Body
• func ExpandVariablesHCLDec(body hcl.Body, spec hcldec.Spec) []hcl.Traversal
• func VariablesHCLDec(body hcl.Body, spec hcldec.Spec) []hcl.Traversal
• type WalkVariablesChild
  • func (c WalkVariablesChild) Body() hcl.Body
• type WalkVariablesNode
  • func WalkExpandVariables(body hcl.Body) WalkVariablesNode
  • func WalkVariables(body hcl.Body) WalkVariablesNode
  • func (n WalkVariablesNode) Visit(schema *hcl.BodySchema) (vars []hcl.Traversal, children []WalkVariablesChild)

Functions

func Expand

func Expand(body hcl.Body, ctx *hcl.EvalContext) hcl.Body

Expand "dynamic" blocks in the given body, returning a new body that has those blocks expanded.

The given EvalContext is used when evaluating "for_each" and "labels" attributes within dynamic blocks, allowing those expressions access to variables and functions beyond the iterator variable created by the iteration.

Expand returns no diagnostics because no blocks are actually expanded until a call to Content or PartialContent on the returned body, which will then expand only the blocks selected by the schema.

"dynamic" blocks are also expanded automatically within nested blocks in the given body, including within other dynamic blocks, thus allowing multi-dimensional iteration. However, it is not possible to dynamically-generate the "dynamic" blocks themselves except through nesting.

parent {
  dynamic "child" {
    for_each = child_objs
    content {
      dynamic "grandchild" {
        for_each = child.value.children
        labels   = [grandchild.key]
        content {
          parent_key = child.key
          value      = grandchild.value
        }
      }
    }
  }
}

func ExpandVariablesHCLDec

func ExpandVariablesHCLDec(body hcl.Body, spec hcldec.Spec) []hcl.Traversal

ExpandVariablesHCLDec is like VariablesHCLDec but it includes only the minimal set of variables required to call Expand, ignoring variables that are referenced only inside normal block contents. See WalkExpandVariables for more information.

func VariablesHCLDec

func VariablesHCLDec(body hcl.Body, spec hcldec.Spec) []hcl.Traversal

VariablesHCLDec is a wrapper around WalkVariables that uses the given hcldec specification to automatically drive the recursive walk through nested blocks in the given body.

This is a drop-in replacement for hcldec.Variables which is able to treat blocks of type "dynamic" in the same special way that dynblock.Expand would, exposing both the variables referenced in the "for_each" and "labels" arguments and variables used in the nested "content" block.

Types

type WalkVariablesChild

type WalkVariablesChild struct {
	BlockTypeName string
	Node          WalkVariablesNode
}

func (WalkVariablesChild) Body

func (c WalkVariablesChild) Body() hcl.Body

Body returns the HCL Body associated with the child node, in case the caller wants to do some sort of inspection of it in order to decide what schema to pass to Visit.

Most implementations should just fetch a fixed schema based on the BlockTypeName field and not access this. Deciding on a schema dynamically based on the body is a strange thing to do and generally necessary only if your caller is already doing other bizarre things with HCL bodies.

type WalkVariablesNode

type WalkVariablesNode struct {
	// contains filtered or unexported fields
}

func WalkExpandVariables

func WalkExpandVariables(body hcl.Body) WalkVariablesNode

WalkExpandVariables is like Variables but it includes only the variables required for successful block expansion, ignoring any variables referenced inside block contents. The result is the minimal set of all variables required for a call to Expand, excluding variables that would only be needed to subsequently call Content or PartialContent on the expanded body.

func WalkVariables

func WalkVariables(body hcl.Body) WalkVariablesNode

WalkVariables begins the recursive process of walking all expressions and nested blocks in the given body and its child bodies while taking into account any "dynamic" blocks.

This function requires that the caller walk through the nested block structure in the given body level-by-level so that an appropriate schema can be provided at each level to inform further processing. This workflow is thus easiest to use for calling applications that have some higher-level schema representation available with which to drive this multi-step process. If your application uses the hcldec package, you may be able to use VariablesHCLDec instead for a more automatic approach.

func (WalkVariablesNode) Visit

func (n WalkVariablesNode) Visit(schema *hcl.BodySchema) (vars []hcl.Traversal, children []WalkVariablesChild)

Visit returns the variable traversals required for any "dynamic" blocks directly in the body associated with this node, and also returns any child nodes that must be visited in order to continue the walk.

Each child node has its associated block type name given in its BlockTypeName field, which the calling application should use to determine the appropriate schema for the content of each child node and pass it to the child node's own Visit method to continue the walk recursively.