README

hclconfig: Configuration file support for cloud-based servers

GoDoc License Build Status (Linux) Coverage Status GoReportCard

Package hclconfig is designed to reduce the effort required to access a configuration file. It is particularly useful for cloud-based server applications that load configuration data via HTTP.

The main features this package provides are:

  • Download configuration via HTTP/HTTPS, from an S3 bucket or from a local file
  • Detect if the configuration file has changed since it was downloaded
  • Provide encryption at rest for confidential information in the configuration file

This package is designed to work with configuration files that are in HCL format. The reason for this choice is that it is straightforward to parse an HCL file into an AST, which makes it possible to implement a convenient mechanism for encrypting and decrypting confidential information.

Simple Example

// eg "https://config.my-app.net/my-app-config.hcl"
// eg "s3://config-bucket/my-app-config.hcl"
// eg "/etc/my-ap-config.hcl"
location := os.Getenv("CONFIG")

// download the config file, and decrypt any confidential information
file, err := hclconfig.Get(location)
exitIfError(err)

var db struct {
    Database struct {
        Provider  string
        SecretDSN string
    }
}

// decode the information we are after into db
err = file.Decode(&db)
exitIfError(err)

db, err := sql.Open(db.Database.Provider, db.Database.SecretDSN)
exitIfError(err)

// simple example of a goroutine that will initiate graceful shutdown
// if it detects a change in the configuration file
go func() {
    for {
        time.Sleep(time.Minute)
        changed, err := file.HasChanged()
        handleErr(err)
        if changed {
            initiateGracefulShutdown()
        }
    }
}()

Encryption

Confidential information is encrypted at rest using AES-256 CBC + HMAC-SHA256.

The 256-bit data encryption key is stored as a ciphertext blob in the configuration file. The data encryption key is encrypted using AWS KMS. Other encryption providers could be implemented in a future version of this package.

Example of an unencrypted configuration file

database {
    provider = "postgres"
    secretDSN = "user=produ password=s3cret dbname=proddb host=prodhost"
}

Example of an encrypted configuration file

database {
    provider = "postgres"

    secretDSN {
        ciphertext = <<END
            RPbAjbNg/2iRsbifmJ3cp4vP8DSM2k6jp7JIFvji3oWjWe50rO5bHFOhMTNfVpTA4T4CBdxJ08
            1AkQOtOFLnj5F1YUzYFbqDW3j3wAvvDgT1lynt5F+DPT/CLQPC0llNKlMbAUAmliChGESdOL4f
            Dw==
            END
    }
}

// data encryption key
encryption {
    // alias/master-kms-key
    kms = <<END
        AQIDAHgLhsBflVB0KoR1VWanrwNzS+ylS6x/KfXjXLqRJA+I1AHRE6ev8Jq+7FsFvelMxsGLAAAAfj
        B8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMhf8Dkptf+b8i
        VKEpAgEQgDusdz5gglVC/aF+15h8majTR8UrdFt3kniu4XHem6NJn4FZCrqVGock5Zd7H96njJgPrJ
        7jhtM7X/st3g==
        END
}

hclconfig CLI utility

A command line utility is included in ./cmd/hclconfig, which makes it easy to encrypt confidential information.

$ hclconfig --help
manage secrets in HCL config files

Usage:
  hclconfig [flags]
  hclconfig [command]

Available Commands:
  encrypt     encrypt secrets in HCL file
  decrypt     decrypt secrets in HCL file
  generate    generate data key for use in HCL config file

Use "hclconfig [command] --help" for more information about a command.

To install the hclconfig CLI:

$ go get -u github.com/jjeffery/hclconfig/cmd/hclconfig

License

MIT

For more information, refer to the Godoc documentation.

Expand ▾ Collapse ▴

Documentation

Overview

Package hclconfig aims to make it easy for cloud-based server programs to access configuration files.

Configuration files can be referenced as a URL or as a local file. Supported URL schemes include http, https, s3 and file. The package provides a mechanism to efficiently determine whether a config file has changed.

It is a common requirement for a configuration file to include sensitive information such as passwords, database connection strings, API keys, and similar. It is poor practice to store this configuration in clear text. This package provides a convenient mechanism for storing sensitive information in a configuration file in encrypted form.

The package is somewhat opinionated. Configuration files are expected to be in HCL format (https://github.com/hashicorp/hcl). Sensitive data is encrypted using a data key, which is stored in the configuration file in encrypted form. Currently AWS KMS is used to encrypt the data key, but other mechanisms could be included in future versions of this package.

The following example shows an HCL configuration file that stores sensitive information.

 // example configuration file
 encryption {
	 // data key encrypted using AWS KMS
	 kms = <<EOF
		AQIDAHgLhsBflVB0KoR1VWanrwNzS+ylS6x/KfXjXLqRJA+I1AGdDZQVyAda6rR1A9A9qT7GAA
		AAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMU28Y
		xot8ipSiVrmZAgEQgDvoJNL7unAdqIgQze98nfCBH0tF3+fbJOeZwjdvI4Od4Loentci39Zjrk
		otk6cofeipCC8UteWQ7lh2Pw==
		EOF
 }

 // storing sensitive information
 database {
	 username = "scott"
	 hostname = "db.example.com"
	 dbname = "production_db"
	 password = {
		 ciphertext = <<EOF
			CDjOKgnBIunYEfUru+jD7OgGmF9+nF3Y
			XsLaJWDe5nIjAYfGdrStPVVYJJdGao0N
			3VFf4bCUFJE=
			EOF
	 }
 }

In this file the `encryption` section has specified a data encryption key that is encrypted using an AWS KMS encryption key. The sensitive data in the configuration file (eg the database password) is encrypted using the data encryption key.

A command line utility `hclconfig` is provided to assist with encrypting and decrypting data in a configuration file. See package "cmd/hclconfig" for details.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type File

type File struct {
	Location     string
	Etag         string
	LastModified time.Time
	Contents     *ast.File
}

File represents a configuration file that has been loaded from a location.

func Get

func Get(location string) (*File, error)

Get downloads the configuration file from the location, parses it and decrypts any sensitive data. The location can be a HTTP/HTTPS URL, an S3 URL, or a local file path.

Example

Code:

// get a config file from a HTTP URL
file, err := Get("https://example.com/config/file.hcl")
if err != nil {
	log.Fatal(err)
}
doSomethingWith(file)

// get a config file from an S3 URL
file, err = Get("s3://bucket-name/config/file.hcl")
if err != nil {
	log.Fatal(err)
}
doSomethingWith(file)

// get a config file from the local filesystem
file, err = Get("./config/file.hcl")
if err != nil {
	log.Fatal(err)
}
doSomethingWith(file)

func (*File) Decode

func (f *File) Decode(v interface{}) error

Decode decodes the contents of the configuration file into the structure pointed to by v.

Example

Code:

file, err := Get("./config.hcl")
if err != nil {
	log.Fatal(err)
}

var config struct {
	Database struct {
		Provider  string
		SecretDSN string
	}
}

if err := file.Decode(&config); err != nil {
	log.Fatal(err)
}

doSomethingWith(config)

/* Config file would look something like:

encryption {
	kms = "<cipher-text-blob>"
}

database {
	provider = "postgresql"
	secretDSN {
		ciphertext = "<cipher-text-blob>"
	}
}

*/	}
}

*/

func (*File) HasChanged

func (f *File) HasChanged() (bool, error)

HasChanged returns true if the config file has changed. It does not download the new contents.

For HTTP(S) and S3 URLs, this function performs a HEAD operation and compares the ETag or the Last-Modified headers. For local files this function performs a file stat and compares the last modified times.

Directories

Path Synopsis
amzn Package amzn contains AWS-specific implementation.
astcrypt Package astcrypt has functions for encrypting and decrypting confidential information in the configuration file AST.
cmd/hclconfig Program hclconfig is a CLI that helps with encrypting HCL config files.
download Package download knows how to download a file from an HTTP URL, an S3 URL or from the local filesystem.
encryption Package encryption performs encryption using AES256 CBC + HMAC SHA256.