helm-envsubst

README

helm-envsubst

Helm plugin to perform variable substitution for your values.yaml files

Feature:

• Substitute System environment variables for ${XXX} placeholders in values.yaml files defined by: -f|--values|--set-file AND the values.yaml file inside the helm chart itself
• Substitute custom environment properties for ${aaa.bbb.ccc} placeholders in helm argument files defined by: -f|--values AND the values.yaml file inside the helm chart itself
• Optionally print the differences to show the variable substitutions
• Embed golang source code into your values.yaml files and have the logic interpreted during install|upgrade without the need for extra scripts
• Define custom go library for use across all values yaml files thus avoiding duplication of embedded go

Documentation

• Install
• Get Started
• Usage
• Examples
• Embedding Go Into Your Custom YAML Files
• Contributing

Install

Prerequisites

• Helm v4.x — this plugin is built against the Helm v4 SDK and is only supported on Helm v4. Check your version with helm version --short; it should report v4.x.x.
• A supported platform — macOS or Linux, on either amd64 (x86_64) or arm64 (aarch64). Windows is not supported.
• git — helm plugin install clones this repository in order to read the plugin manifest and run its install hook.
• curl or wget — the install hook uses whichever is available to download the pre-built binary. macOS and mainstream Linux distributions ship at least one by default.

You do not need Go installed: the install downloads a pre-built, statically-linked binary — nothing is compiled locally.

Install the plugin

From any directory (the command installs into Helm's own plugin directory, so your current working directory does not matter), run:

helm plugin install https://github.com/myuseridbws/helm-envsubst --verify=false

--verify=false is required: Helm v4 verifies plugins on install by default, and this is an unsigned, legacy-format plugin. This only skips Helm's plugin-signature check — the install hook still verifies the downloaded binary against its published SHA-256 checksum (step 3 below). Signing the releases is a possible future enhancement.

This single command:

1. Clones this repository into Helm's plugin directory — the path reported by helm env HELM_PLUGINS.
2. Reads plugin.yaml and runs the install hook, install-binary.sh.
3. The hook detects your OS and architecture, downloads the matching release tarball from the GitHub Pages site (https://myuseridbws.github.io/my-helm-charts/releases/helm-envsubst/<version>/), verifies its SHA-256 checksum, and extracts the hsubst binary into the plugin directory.

The version installed is whatever plugin.yaml on the default branch currently declares.

• NOTE: When a tag whose name starts with a v is pushed (e.g. v1.0.0) AND the tag's version (with the v stripped) matches the version in plugin.yaml, the release workflow (.github/workflows/release.yml) runs automatically: it builds the platform binaries and their corresponding sha256 checksum file and pushes them to the my-helm-charts GitHub Pages repo that this helm plugin install command downloads from.

Verify the installation

Confirm the plugin is registered:

helm plugin list

You should see an envsubst row — for example:

NAME      VERSION  TYPE    APIVERSION  PROVENANCE  SOURCE
envsubst  0.0.1    cli/v1  legacy      unknown     unknown

The plugin is now invoked by inserting envsubst between helm and a subcommand (install / upgrade / template) — see Get Started and Examples below.

Uninstall

To remove the plugin:

helm plugin uninstall envsubst

Get Started

This plugin is very easy to use.

Instead of running:

helm install foobar my-chart.tgz -f my-custom.yaml

You simply add the plugin name in between 'helm' and 'install' and run as such:

helm envsubst install foobar my-chart.tgz -f my-custom.yaml

All other helm arguments are the same.

Doing this will substitute any ${XXX} placholder in 'my-custom.yaml' if there is a matching System environment variable.

This behaves similarly to the Unix envsubst command with the exception that if the System environment variable does not exist, the placeholder remains untouched.

The plugin will then invoke helm using the file with the substituted values instead of the original file (with the placeholders).

For additional options see the Usage section

Usage

helm envsubst [OPTIONS] install|upgrade|etc. <normal-helm-arguments>

'envsubst' plugin OPTIONS:
==========================
--env-values <file> [OPTIONAL] A yaml file containing values that can be substituted for \${placeholder}
                    variables in one or more of the helm values files specified on the remaining helm
                    command line.

--keep-tmp-files    [OPTIONAL] Retains the temporary files that were used as the target for the 
                    substituted vars.  The default behavior is to delete the files after the 
                    underlying helm command is finished.

--print-diff        [OPTIONAL] Print the diff between the original file and the file with the
                    injected values.  The default is not to print.

--custom-go         [OPTIONAL] A golang source file whose functions will be made available to be
                    referenced as embedded go in all custom yaml files provided on the helm
                    command line.

Special Handling of Helm Command Line Arguments:
================================================
The following helm command line arguments invoke special behavior via the envsubst plugin.
The referenced files are 'pre-processed' in order to replace variable placeholders with actual
values.  The placeholders can be of the form:  ${AAA}, ${bbb}, ${aaa.bbb.ccc}, etc.

-f|--values <file>  [OPTIONAL] Substitute "dollar curly" variables with non-null variable values.
                    If no value is found, then the variable is left alone to be resolved during install
                    or runtime.
                    NOTE:  If a file whose basename is "env-values.yaml" is specified, this file will be
                    used as the primary source for resolving the "dollar curly" variables in the other
                    helm values files.

--set-file  <helm-variable>=<file>
                    [OPTIONAL] Substitute "dollar curly" variables with non-null variable values.
                    If no value is found, then the variable is left alone to be resolved during
                    install or runtime.

--set-file-encoded <helm-variable>=<file>
                    [OPTIONAL] Set the value of a helm variable to the base64 encoded contents of
                    the file.  Also, before encoding the file, the "dollar curly" variables will be
                    substituted if a value exists for the variable.  This argument is a custom
                    helm argument.  It is not one of the helm standard arguments.
                    NOTE: The encoded file will be appended at the end of the helm command using a
                    "--set <variable-name>=<encoded-string> argument.  This may cause issues if the
                    order prcedence of the <variable-name> is significant since it will overwrite
                    another value if one was set before it.

The values are substituted for the placeholders based on whether or not a matching variable
exists in either the '--env-values' yaml file (if one is provided) or the system environment variables.

Examples

The simplest use case is when you supply no arguments to your helm install|upgrade as such:

helm envsubst install my-release my-helm-chart.tgz

By invoking the install using envsubst, the plugin will automatically perform variable substitution for ${...} placeholders in the chart's embedded values.yaml file.

If the chart defines a values.yaml file with some general environment properties (for example: ${DOCKER_REPO_HOST}), these properties could be set as system environment settings (such as Jenkins environment variables) and automatically provided as the substituted value for the chart.

Again....without the need to specify any -f custom-values.yaml argument.

Other examples:

Example 1 - Simple System Environment Substitution Only

helm envsubst install my-release . -f ./my-values.yaml

where:

export LANGUAGE=en_US

./my-values.yaml:

myHelmVariable1: ${foo.bar}
myHelmVariable2: ${blech}
language: ${LANGUAGE}

Resulting YAML file used for helm chart:

myHelmVariable1: ${foo.bar}
myHelmVariable2: ${blech}
language: en_US

Example 2 - Use YAML File For Substitution

helm envsubst --env-values ~/my-env.yaml install my-release . -f ./my-values.yaml

where:

./my-env.yaml:

foo:
  bar: "abcdefg"
not-used: "123456"

./my-values.yaml:

myHelmVariable1: ${foo.bar}
myHelmVariable2: ${blech}
installedBy: ${USER}

Given the above command line and files, the actual values.yaml file that is used by helm during the install woudl be the following (having substituted the known variables for the placeholders):

./tmp-file-with-substituted-values.yaml:

myHelmVariable1: abcdefg
myHelmVariable2: ${blech}
installedBy: admin

NOTE: Unlike the Unix 'envsubst' command, variables that are NOT matched remain in tact instead of resolving to an empty string or null. This is why ${blech} has remained the same and is not empty string or null.

Example 3 - Using --set-file and --set-file-encoded

helm envsubst --env-values my-env.yaml --keep-tmp-files install my-release . -f ./my-values.yaml --set-file foo.bar=./my-other-values.yaml --set-file blech=./my-other-values2.yaml

helm envsubst --print-diff install my-release . --values ./xyz.yaml --set-file-encoded my.data.encoded=./my-values.yaml

• NOTE: When using the --set-file-encoded argument as shown above, it would be common to use it in conjunction with the helm b64dec utility.

For example, if your helm chart has the following ConfigMap defined:

kind: ConfigMap
metadata:
  name: test-config-map
data:
  test2.yaml:
{{ .Values.my.data.encoded | b64dec | indent 4 }}

Then running the above command would take the file ./my-values.yaml, substitute its values and then basically inject the files substituted contents into the config map as such:

kind: ConfigMap
metadata:
  name: test-config-map
data:
  test2.yaml:
    myHelmVariable1: abcdefg
    myHelmVariable2: ${blech}
    installedBy: admin

Embedding Go Into Your Custom YAML Files

This plugin has the ability to embed golang syntax and logic into your custom values files. This is a powerful way to control the values used to install your application that may be dynamic in nature.

In order to embed your go code into your yaml file, you need to follow a specific syntax which lets the plugin know where the go statements start and stop.

Here is an example:

##<go>
##import "strings"
##MAIN
embedded:
##if "EN_US" == strings.ToUpper("${LANGUAGE}") {
  greeting: "Hello World!"
##} else {
  greeting: "Hola Mundo!"
##}
  language: "${LANGUAGE}"
##</go>

As you can see, the "code block(s)" of go to be interpreted are designated using the following begin and end markers:

##<go>
...
##</go>

Within the go code block, all lines that start with ## will be interpreted.

• NOTE: The ## can be preceded by any number of spaces in order to preserve a proper indentation for readability purposes

Standard go libraries are available. You may not be able to access unsafe and/or other remote libraries. This is a function of the interpreter.

You may declare and reference functions in between ##<go> and ##MAIN.

The section between ##MAIN and ##</go> is the declarative section of the interpreted go and you can mix go logic (such as conditionals, loops, etc.) and YAML to affect the final YAML content.

So...using the example above, this would be the substituted values.yaml file used during the helm install:

Given the following system environment variable:

export LANGUAGE=en_US

embedded:
  greeting: Hello World!
  language: en_US

Example - Using --custom-go

helm envsubst --custom-go ./sample.go install my-release . -f ./my-values.yaml

where:

./sample.go:

package  customgo
import (
        "io/ioutil"
        "log"
        "net/http"
)
func testRestGet() string {
        resp, err := http.Get("https://httpbin.org/get")
        if err != nil {
                log.Fatalln(err)
        }
        defer resp.Body.Close()

        body, err := ioutil.ReadAll(resp.Body)
        if err != nil {
                log.Fatalln(err)
        }
        return string(body)
}

./my-values.yaml:

aaa: ${foo.bar}
bbb: ${blech}
ccc: ${abc.xyz}
##<go>
##fmt.Println("Calling remote REST API:")
##fmt.Println("========================")
##fmt.Println(customgo.testRestGet())
##</go>

During helm install, the following would be printed to output:

Calling remote REST API:
========================
{
  "args": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Host": "httpbin.org",
    "User-Agent": "Go-http-client/2.0",
    "X-Amzn-Trace-Id": "Root=1-5ef7bfbe-a8c5594e962c6adc868070c2"
  },
  "origin": "193.43.244.125",
  "url": "https://httpbin.org/get"
}

• NOTE: This example is just to demonstrate the way some go source file could be invoked from within the custom yaml file.

License

Apache-2.0

Contributing

This is my first attempt at sharing some utiilty with the general public.

I would like to acknowledge and thank the following projects from which I obtained some of the build/infrastructure code for building helm plugins:

• https://github.com/lrills/helm-unittest

• https://github.com/quintush/helm-unittest