README ¶
runiac - Run IaC Anywhere With Ease
A tool for running infrastructure as code (e.g. Terraform) anywhere with ease.
- Ability to change and test infrastructure changes locally with a production like environment
- Ability to make infrastructure changes without making pipeline changes
- Quality developer experience
- Container-based, execute anywhere and on any CI/CD system
- Multi-Region deployments built-in
- Handling groups of regions for data privacy regulations
- Enabling "terraservices"
- Keeping Your Pipelines Simple
- Plugin-based
runiac's primary goal is enabling easy, meaningful local development that mimics a production deployment.
This enables two large benefits:
- Changes can be tested quickly and reliably from a local machine, knowing the
We'd love to hear from you! Submit github issues for questions, issues or feedback.
Table of Contents generated with DocToc
- How does runiac work?
- Demo
- Install
- Tutorial
- Terminology
- Using runiac
- Runners
- Contributing
How does runiac work?
runiac's primary goal is to enable developers to spend more time iterating on valuable infrastructure changes rather than pipeline or glue code.
It enables this by following the smart endpoints, dumb pipelines
, portability
, and fun
principles defined at doeac
.
- Infrastructure changes do not require pipeline changes
- Ability to change and test infrastructure changes locally with a production like environment
What expo did for react native development, runiac does for terraform. What webpack did for react development, runiac does for terraform.
- Directory layout
- Steps
- Primary deployment type
- Regional
- Config
- Primary Regions
- Regional Regions
- Project
- Environment
- Account
- Namespace
--local
- Tracks
Demo
- Record Gif Here of running
runiac
Install
homebrew tap:
brew install optum/tap/runiac
manually:
Download the pre-compiled binaries from the releases page and copy to the desired location.
Tutorial
For more detailed examples of runiac, be sure to check out the examples directory!
Terminology
Steps
- Steps follow a folder naming convention of
step{progressionLevel}_{stepName}
- A Step's Progression Level identifies the ordering of execution.
- All steps receive a common set of input variables (see below)
- All steps receive the output variables of the steps in the progression level ahead of them.
- For example:
- output variables of
step1
will be sent intostep2
- output variables of
step1
andstep2
will be passed intostep3
- so on, so forth
- output variables of
- For example:
- Steps will automatically execute tests after a successful deployment, these are primarily used for smoke testing (see below)
- Steps have two types of deployment,
Primary
andRegional
.
Step Execution Examples
In the following Track directories:
tracks/iamsso
├── step1_aws
└── step1_onprem_adgroups
step1_aws
and step1_onprem_adgroups
will be executed at concurrently at the same time.
tracks/network
├── step1_vpc
└── step2_egress_proxy
step1_vpc
will be executed first and after completion step2_egress_proxy
will be executed.
tracks/network
├── step1_vpc
├── step1_aws
└── step2_cool_step
└── step2_special_step
└── step3_something_awesome
step1_vpc
andstep1_aws
will be executed first concurrently at the same time.- After completion of both
step2_cool_step
andstep2_special_step
will be executed concurrently at the same time. a. The output variables of bothstep1
's will be passed intostep2
steps - After completion of both
step2
's,step3_something_awesome
will be executed. a. The output variables of allstep1
andstep2
's will be passed intostep3
steps
Step Deployment Types
Step deployment types facilitate multi-region deployments. runiac will first execute every primary step deployment type in a track.
If the primary region deployment is successful, runiac will then run each step's regional deployment type (regional
) concurrently across each region defined in regional_regions
.
Primary deployments represent all iac in the top level directory of the executing step.
Currently, the primary deployment type is executed once per region group.
For example, in the us
region group, the primary code would only be executed in the primary region of the us
region group, us-east-1
.
tracks/network
├── step1_vpc
├──--- *.tf
Regional deployments represent iac in the regional
directory of the executing step. This code will be executed concurrently N
times based on N
count of regions defined in regional_regions
configuration.
tracks/network
├── step1_vpc
├──--- regional
├──--------*.tf
Tracks
- All Tracks beside the pre-track will be executed in parallel
- By definition tracks do not have dependencies on each other. If they do, treat them as one track with multiple steps
- For a track to be executed, at least one Step has to be defined within it
Default Track
For projects that are relatively straightforward and don't require multiple tracks, you do not need to use tracks in your folder heiarachy. The benefit of this approach is a simpler directory hierarchy, and you still have the possibility to scale out with multiple tracks down the road.
Be aware that with this simpler approach, you cannot use pre-tracks (see below).
A sample structure from the root directory of your project might look like this:
step1_sample/
step1_another_one/
runiac.yml
To whitelist steps for execution when using a default track, use the track name default
in your step_whitelist
:
RUNIAC_STEP_WHITELIST="#runiac#default#sample,#runiac#default#another_one"
Pre-track
A pre-track is a track that runs before all other tracks. After this track completes, the remaining tracks are executed in parallel. If the pre-track execution fails, no other tracks will be attempted. To create a pre-track, create a directory called _pretrack
in the tracks
directory.
Using runiac
To use runiac to deploy your infrastructure as code, you will need:
Docker
installed locallyruniac
installed locally
Inputs
Configuration for executing runiac is done through environment variables. For a list of options, see the code here.
Choosing which steps to execute
By default, steps will not be executed unless they are explicitly configured to do so.
runiac_STEP_WHITELIST
: list of step names to include in execution
When providing a list of steps to execute using the runiac_STEP_WHITELIST
environment variable, the general syntax is as follows:
#PROJECT[#TRACK]#STEP_NAME,...
Where:
PROJECT
is the value of theruniac_PROJECT
environment variable (default toruniac
is not specified)[TRACK]
is the name of the track the step is located under (unless using the default track)STEP_NAME
: is the name of the step, without the leadingstepX_
prefix
For example, given the following runiac directory setup:
tracks/
├── infra/
├──── step1_sample/
├── shared
├──── step1_sample/
├──── step1_another_one/
If you wanted all three steps to be executed, you would specify them as such:
runiac_STEP_WHITELIST="#runiac#infra#sample,#runiac#shared#sample,#runiac#shared#another_one"
A configuration file can exist in either a track's or step's directory.
runiac.yaml
enabled: <true|false> # This determines whether the step will be executed
execute_when: # This will conduct a runtime evaluation on whether the step should be executed
region_in: # By matching the `var.region` input variable
- "region-1"
Versioning
The most flexible way to specify a version string for your deployment artifacts is to use the VERSION
environment variable. You
can source your version string however you wish with this approach.
Otherwise, you can create a version.json
file at the root of the directory structure, with a version
element:
{
"version": "v0.0.1"
}
If both are present, version.json
takes precedence over the VERSION
environment variable.
Provider Plugin Caching
runiac uses provider plugin caching. Projects that use runiac are responsible for creating the directories that are used for provider caching and also creating their own .terraformrc file. Please note that with the upgrade to Terraform v0.13
, projects will need to update their filesystem layout for local copies of providers as stated here.
Runners
Terraform
Using Previous Step Output Variables
By default, runiac will pass in the output variables from previous steps into the current step.
For example, if step1_s3_bucket
has a defined outputs.tf
:
output "producer_assume_role_arn" {
description = "The name of the S3 Bucket (Purple Bucket) that will be the destination of all logs on the AWS account."
value = aws_iam_role.producer_lambda_role.arn
}
Then step2_logging
can use it by declaring a variable as {step_name}-{output_variable_name}
, for example:
variable "s3_bucket-producer_assume_role_arn" {
type = string
description = "Variable from step1_s3_bucket"
}
If a pre-track exists, all the step output variables from the pre-track will be available as input variables to other steps. These pre-track output variables can be used in other steps by declaring a variable as pretrack-{pretrack_step_name}-{output_variable_name}
. For example:
variable pretrack-project_creation-project_name {
type = string
description = "Variable from the project_creation step in the pre-track"
}
When working in a regional context, additional passed variables are available from prior step's regional deployments.
NOTE: Only output variables from primary and same region regional deployments are available to regional deployments
To access previous step's regional output variables one can do so by adding -regional-
after the step name of the variable, like the following:
variable "s3_bucket-regional-producer_assume_role_arn" {
type = string
description = "Variable from step1_s3_bucket"
}
If a pre-track exists, you can also access the regional output variables from the pre-track steps by declaring a variable as pretrack-{pretrack_step_name}-regional-{output_variable_name}
. For example:
variable pretrack-resource_groups-regional-resource_group_name {
type = string
description = "Variable from the resource_groups regional step in the pre-track"
}
Common Input Variables
variable "runiac_region" {
type = string
description = "The region for this Terraform run"
}
variable "runiac_namespace" {
type = string
description = "The namespace for this Terraform run" # During PR, this value is set to `pr-{changeId}`, ie. pr-3
}
variable "runiac_environment" {
type = string
description = "Designates the production level of the associated resource" # During PR, this value is set to `pr`
}
variable "runiac_app_version" {
type = string
description = "Designates the specific version of the application deployment"
}
variable "runiac_account_id" {
type = string
description = "ID of the Account being deployed to"
}
variable "runiac_core_account_ids_map" {
type = map(string)
description = "Mapping of the available core account ids, AWS: logging_final_destination, guard_duty_master, logging_bridge_aws, logging_bridge_azu"
}
### Optional Variables
# The initial use case for this variable is to know which account is the original target after overriding
# provider.assume_role.arn in terraform.
variable "runiac_target_account_id" {
type = string
description = "The account id that the step function told the fargate task to deploy to"
}
variable "runiac_deployment_ring" {
type = string
description = "The deployment ring currently being executed in"
}
variable "runiac_stage" {
type = string
description = "The stage currently being executed in"
}
variable "runiac_track" {
type = string
description = "The track currently being executed in"
}
variable "runiac_step" {
type = string
description = "The step currently being executed in"
}
variable "runiac_region_deploy_type" {
type = string
description = "The step deployment type, either primary or regional"
}
variable "runiac_region_group" {
type = string
description = "The region group being deployed in, supported: 'us'"
}
variable "runiac_primary_region" {
type = string
description = "The primary region of the runiac_region_group"
}
variable "runiac_region_group_regions" {
type = string
description = "The list of regions within the runiac_region_group. This list represents the regions that will be used to execute the `regional` directory within a step."
}
Tests
Tests within a step will automatically be executed after a successful deployment.
Test Convention Requirements
- Need to be defined in a
tests
directory within the step's directory. - Need to be golang tests OR compiled to an executable named
tests.test
- If using golang tests, runiac Build Container will compile the tests to an executable automatically as part of container build process
- Golang tests are the recommendation (ie. Terratest).
- The tests directory will receive the terraform outputs of the step as
TF_VAR
environment variables
For example in the following source code directory:
tracks/iamsso/step1_aws/
├── tests
│ └── step_test.go
├── backend.tf
├── outputs.tf
├── providers.tf
├── read_only_role.tf
├── shared.tf
├── variables.tf
└── versions.tf
As part of container build, runiac will compile the golang test code into:
tracks/iamsso/step1_aws/
├── tests
│ └── tests.test
├── backend.tf
├── outputs.tf
├── providers.tf
├── read_only_role.tf
├── shared.tf
├── variables.tf
└── versions.tf
runiac will then execute tests.test
after a successful step deployment.
Conventions and Supported Configurations
Backend
By convention the backend type will be automatically configured.
Supported Types:
- S3
- AzureRM
- GCS
- Local
If defining local, terraform will be executed "fresh" each time. This works very well when the step is only executing scripts/binaries through local-exec
.
While you normally cannot use variable interpolation in typical Terraform backend configurations, runiac allows you some more flexibility in this area. Depending on which backend provider you are intending to use, the sections below detail which variables can be used in your configuration. These variables will be interpolated by runiac itself prior to executing Terraform.
Supported variables for dynamic key
, bucket
or role_arn
configuration:
${var.runiac_region_deploy_type}
: required inkey
${var.region}
: required inkey
${var.runiac_step}
${var.core_account_ids_map}
${var.runiac_target_account_id}
${var.runiac_deployment_ring}
${var.environment}
${local.namespace-}
(temporary backwards compatibility variable)
Example Usage:
terraform {
backend "s3" {
key = "${var.runiac_target_account_id}/${local.namespace-}${var.runiac_step}/${var.runiac_region_deploy_type}-${var.region}.tfstate"
bucket = "product-tfstate-${var.core_account_ids_map.runiac_deploy}"
role_arn = "arn:aws:iam::${var.core_account_ids_map.runiac_deploy}:role/StateRole"
acl = "bucket-owner-full-control"
region = "us-east-1"
encrypt = true
}
}
Supported variables for dynamic bucket and/or prefix
configuration:
${var.gaia_region_deploy_type}
${var.region}
${var.gaia_step}
${var.core_account_ids_map}
${var.gaia_target_account_id}
${var.gaia_deployment_ring}
${var.environment}
${local.namespace-}
(temporary backwards compatibility variable)
Example Usage:
terraform {
backend "gcs" {
bucket = "df-${var.environment}-tfstate"
prefix = "infra/${var.gaia_deployment_ring}/${var.gaia_region_deploy_type}/${var.region}/${local.namespace-}infra.tfstate"
}
}
Provider (AWS)
At this time, providers must be defined in a providers.tf
file for this configuration to work
By convention runiac will assume role into the OrganizationAccountAccessRole
of the ACCOUNT_ID
environment variable prior to executing any steps. However, sometimes there is value in explicitly defining terraform infrastructure for multiple accounts in the same repository, for example a subset of "Core" accounts all other accounts share. To support this, Bedrock container will use the provider.assume_role.role_arn
value in the step's providers.tf
where one can explicitly set which account the terraform will be executed in via the assume role arn.
Provider (Azurerm)
At this time, providers must be defined in a providers.tf
file for this configuration to work
To mirror the assume_role
functionality for AWS core deployments in runiac, Azure supports deploying to Azure core accounts using the subscription_id
field in the provider. For example:
provider "azurerm" {
version = "1.35.0"
subscription_id = var.core_account_ids_map.tenant_core_azu
}
When using this functionality, you can only specifiy an account in the core_account_ids_map
Terraform variable. If this value is not specified, runiac will deploy to the account that was specified by the ARM_SUBSCRIPTION_ID
environment variable.
The runiac_target_account_id
and any key available in the core_account_ids_map
input variable can be used in the provider.assume_role.role_arn
value. For example, to deploy infrastructure only the AWS Bridge Logging accounts the configuration would minimally need to include:
provider "aws" {
assume_role = {
role_arn = "arn:aws:iam::${var.core_account_ids_map.logging_bridge_aws}:role/OrganizationAccountAccessRole"
}
}
In this example the terraform creds_id
and account_id
input variables will match the values for logging_bridge_aws
NOTE: Only using the
OrganizationAccountAccessRole
role is supported
Working with Secrets
Secrets within runiac should be stored within AWS SSM Parameter Store as encrypted parameters. runiac utilizes a naming hierarchy for scoping the secrets.
This hierarchy goes as /bedrock/delivery/{csp}/{stage}/{track}/{step}/{ring}/param-{parameter}
Parameters defined at a more granular level (ie. step over csp) will take precedent.
See the appropriate unit tests for how this works here.
NOTE: These parameters must be defined in the account Bedrock delivery framework executes in.
Deployment Ring Specific Configurations
The most common and terraform friendly to implement deployment specific configuration is via count
and simple if
statements in the terraform code based on the passed in var.runiac_deployment_ring
value.
Override Files
The alternative option is using terraform's override feature. runiac handles this based on the override
directory within a step.
The supported override files are below:
override.tf
- file will be added for all deployment rings and deployments, including Self-Destroy.ring_*ring-name*_override.tf
- file will be added for the specified deployment ring and deployments, including Self-Destroy.destroy_override.tf
- file will be added for all deployment rings and Self-Destroy deployments.destroy_ring_*ring-name*_override.tf
- file will be added for the specified deployment ring and Self-Destroy deployments.
For example, in the following step when deploying to:
local
deployment ring: thering_local_override.tf
file will be added to the executed terraformprod
deployment ring: thering_prod_override.tf
file will be added to the executed terraform
tracks/iamsso/step1_aws/
├── override
│ └── ring_local_override.tf
│ └── ring_prod_override.tf
├── backend.tf
├── outputs.tf
├── providers.tf
├── read_only_role.tf
├── shared.tf
├── variables.tf
└── versions.tf
For example, in main.tf
:
# super important resource that cannot be deleted
resource "aws_s3_bucket" "centralized_logging_master_bucket" {
bucket = "log-compliance-data"
lifecycle {
prevent_destroy = true
}
}
And then for ephemeral environments (e.g. local development), ring_local_override.tf
:
# super important resource that can be deleted in local deployment ring
resource "aws_s3_bucket" "centralized_logging_master_bucket" {
lifecycle {
prevent_destroy = false
}
}
This has the benefit of not introducing the subtle complexities of "toggling" between two different resources with count
NOTE: Terraform recommends using this feature sparingly as it is not noticeable the value is overridden in the main terraform files.
A common use case for this feature is controlling terraform lifecycle
parameters for ephemeral environments while keeping the main terraform files defined for production.
Contributing
Please read CONTRIBUTING.md first.
Running Locally
runiac is only executed locally with unit tests. To execute runiac child projects locally, one would need to build this container first.
Docker Build:
$ DOCKER_BUILDKIT=1 docker build -t runiac .
We recommend adding an alias to install the cli locally:
alias runiacdev='(cd <LOCAL_PROJECT_LOCATION>/cmd/cli && go build -o $GOPATH/bin/runiac) && runiac'
This allows one to use the the examples
for iterating on runiac changes.
$ cd examples/terraform-gcp-hello-world
$ runiacdev -a <YOUR_GCP_PROJECT_ID> -e nonprod --local
Directories ¶
Path | Synopsis |
---|---|
cmd
|
|
Package mocks is a generated GoMock package.
|
Package mocks is a generated GoMock package. |
pkg
|
|
plugins
|
|
terraform/pkg/terraform
Package terraform allows to interact with Terraform.
|
Package terraform allows to interact with Terraform. |