terracost

README

TerraCost

Go library for estimating Terraform costs using ingested cloud vendor prices. It is meant to be imported and used by programs (API's or standalone) with access to a MySQL-compatible database and the Internet.

Installation

go mod edit -replace github.com/hashicorp/terraform=github.com/cycloidio/terraform@v1.1.9-cy && go get github.com/cycloidio/terracost

We need to do a -replace because of https://github.com/golang/go/issues/30354#issuecomment-466557015. We have a custom fork of Terraform and in order to use TerraCost it needs to be replaced when importing also.

Requirements

• Go 1.16 or newer
• MySQL database

Provider support

Currently Terracost supports the following providers and with an specific subset of resources from them:

• AWS
• Google
• AzureRM

Google Credentials

To be able to ingest the pricing data from Google the credentials needed have to have access to Compute Engine API and have also billing enabled. This is needed to be able to fetch the Machine Types, for the SKUs we would only need an normal set of credentials (or even API Key) but as we need both we have to use the more complex one. If you do not know how to activate those permissions, just use the credentials and the import will fail and on the logs will inform of what is missing and how to activate it.

Usage

Migrating the database

db, err := sql.Open("mysql", "root:password@tcp(IP:3306)/databasename?multiStatements=true")

// Can be called on every start of your program, it does nothing if the migrations
// have been executed already.
err = mysql.Migrate(context.Background(), db, "pricing_migrations")

Ingesting pricing data

db, err := sql.Open("mysql", "root:password@tcp(IP:3306)/databasename?multiStatements=true")
backend := mysql.NewBackend(db)

// service can be "AmazonEC2" or "AmazonRDS"
// region is any AWS region, e.g. "us-east-1" or "eu-west-3"
ingester, err := aws.NewIngester(service, region)
err = terracost.IngestPricing(context.Background(), backend, ingester)

Tracking ingestion progress

We're using the github.com/machinebox/progress library for tracking ingestion progress.

1. Create a channel that will receive progress updates and set up a goroutine (it will print the bytes ingested out of bytes total and remaining time each time progress update is sent on the channel):

progressCh := make(chan progress.Progress, 0)

go func() {
	for p := range progressCh {
		// Check the docs for all available methods: https://pkg.go.dev/github.com/machinebox/progress#Progress
		fmt.Printf("%d / %d (%s remaining)\n", p.N(), p.Size(), p.Remaining().String())
	}
}()

1. Initialize an ingester capable of tracking progress (in this example the channel will receive an update every 5 seconds):

ingester, err := aws.NewIngester(service, region, aws.WithProgress(progressCh, 5*time.Second))

1. Use the ingester as in the previous section.

Estimating a Terraform plan

Plan estimation is possible after all the relevant pricing data have been ingested and stored in the database.

1. Generate a plan using terraform and convert it to JSON:

terraform plan -out update.tfplan
terraform show -json update.tfplan > tfplan.json

1. Read the plan file, estimate it and show the resource differences:

db, err := db.Open("mysql", "...")
backend := mysql.NewBackend(db)

file, err := os.Open("path/to/tfplan.json")
plan, err := terracost.EstimateTerraformPlan(context.Background(), backend, file)

for _, res := range plan.ResourceDifferences() {
  priorCost, err := res.PriorCost()
  plannedCost, err := res.PlannedCost()

  fmt.Printf("%s: %s -> %s\n", res.Address, priorCost, plannedCost)
}

Check the documentation for all available fields.

Usage estimation

Some resources do cannot be estimated just by the configuration and need some extra usage information, for that we have some default on usage/usage.go which are also all the resources and options we support currently and can be overwritten when estimating if passing a custom one instead of the custom Default one.

Examples

For more examples, please check examples.

Contributing

For Contributing Guide, please read CONTIBUTING.md.

License

This project is licensed under the MIT license. Please see LICENSE for more details.

Meet Cycloid

Cycloid is a hybrid cloud DevOps collaboration platform providing end-to-end frameworks to accelerate and industrialize software delivery.

As of now, we have three open-source tools:

• TerraCognita: Read from your existing cloud providers and generate IaC in Terraform
• InfraMap: Reads .tfstate or HCL to generate a graph specific for each provider
• TerraCost: Cloud cost estimation for Terraform in the CLI

...and the functionality of each is also embedded in our DevOps solution, which you can find out more about here.

Documentation

Overview

Package terracost provides functionality to estimate the costs of infrastructure based on Terrafom plan files.

This package depends on the pricing data located in a MySQL database to work correctly. The following snippet will run all required database migrations and ingest pricing data from AmazonEC2 in eu-west-3 region:

db, err := sql.Open("mysql", "...")
backend := mysql.NewBackend(db)

// Run all database migrations
err = mysql.Migrate(ctx, db, "pricing_migrations")

// Ingest pricing data into the database
ingester := aws.NewIngester("AmazonEC2", "eu-west-3")
err = terracost.IngestPricing(ctx, backend, ingester)

With pricing data in the database, a Terraform plan can be read and estimated:

file, err := os.Open("path/to/tfplan.json")
plan, err := terracost.EstimateTerraformPlan(ctx, backend, file)

for _, res := range plan.ResourceDifferences() {
    fmt.Printf("%s: %s -> %s\n", res.Address, res.PriorCost().String(), res.PlannedCost().String())
}

Index

• func EstimateHCL(ctx context.Context, be backend.Backend, afs afero.Fs, ...) ([]*cost.Plan, error)
• func EstimateTerraformPlan(ctx context.Context, be backend.Backend, plan io.Reader, u usage.Usage, ...) (*cost.Plan, error)
• func IngestPricing(ctx context.Context, be backend.Backend, ingester Ingester) error
• type Ingester

Functions

func EstimateHCL

func EstimateHCL(ctx context.Context, be backend.Backend, afs afero.Fs, stackPath, modulePath string, ftg bool, u usage.Usage, providerInitializers ...terraform.ProviderInitializer) ([]*cost.Plan, error)

EstimateHCL is a helper function that recursively reads Terraform modules from a directory at the given stackPath and generates a planned cost.State that is returned wrapped in a cost.Plan. It uses the Backend to retrieve the pricing data. The modulePath is used to know if the module is not defined on the root of the stack, If Force Terragrunt(ftg) is set then we'll just run Terragrunt

func EstimateTerraformPlan

func EstimateTerraformPlan(ctx context.Context, be backend.Backend, plan io.Reader, u usage.Usage, providerInitializers ...terraform.ProviderInitializer) (*cost.Plan, error)

EstimateTerraformPlan is a helper function that reads a Terraform plan using the provided io.Reader, generates the prior and planned cost.State, and then creates a cost.Plan from them that is returned. It uses the Backend to retrieve the pricing data.

func IngestPricing

func IngestPricing(ctx context.Context, be backend.Backend, ingester Ingester) error

IngestPricing uses the Ingester to load the pricing data and stores it into the Backend.

Types

type Ingester

type Ingester interface {
	// Ingest downloads pricing data from a cloud provider and sends prices with their associated products
	// on the returned channel.
	Ingest(ctx context.Context, chSize int) <-chan *price.WithProduct

	// Err returns any potential error.
	Err() error
}

Ingester represents a vendor-specific mechanism to load pricing data.