terraform-provider-databricks

README

Databricks Terraform Provider

AWS tutorial | Azure tutorial | End-to-end tutorial | Authentication | Troubleshooting Guide | Changelog | Contributing and Development Guidelines

Databricks Terraform provider works with Terraform 1.1.5 or newer. To use it please refer to instructions specified at registry page:

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}

If you want to build it from sources, please refer to contributing guidelines.

Then create a small sample file, named main.tf with approximately following contents. Replace <your PAT token> with newly created PAT Token.

provider "databricks" {
  host  = "https://abc-defg-024.cloud.databricks.com/"
  token = "<your PAT token>"
}

data "databricks_current_user" "me" {}
data "databricks_spark_version" "latest" {}
data "databricks_node_type" "smallest" {
  local_disk = true
}

resource "databricks_notebook" "this" {
  path     = "${data.databricks_current_user.me.home}/Terraform"
  language = "PYTHON"
  content_base64 = base64encode(<<-EOT
    # created from ${abspath(path.module)}
    display(spark.range(10))
    EOT
  )
}

resource "databricks_job" "this" {
  name = "Terraform Demo (${data.databricks_current_user.me.alphanumeric})"

  new_cluster {
    num_workers   = 1
    spark_version = data.databricks_spark_version.latest.id
    node_type_id  = data.databricks_node_type.smallest.id
  }

  notebook_task {
    notebook_path = databricks_notebook.this.path
  }
}

output "notebook_url" {
  value = databricks_notebook.this.url
}

output "job_url" {
  value = databricks_job.this.url
}

Then run terraform init then terraform apply to apply the hcl code to your Databricks workspace.

OpenTofu Support

OpenTofu is an open-source fork of Terraform with the MPL 2.0 license. The Databricks Terraform provider should be compatible with OpenTofu, but this integration is not actively tested and should be considered experimental. Please raise a Github issue if you find any incompatibility.

Switching from databrickslabs to databricks namespace

To make Databricks Terraform Provider generally available, we've moved it from https://github.com/databrickslabs to https://github.com/databricks. We've worked closely with the Terraform Registry team at Hashicorp to ensure a smooth migration. Existing terraform deployments continue to work as expected without any action from your side. We ask you to replace databrickslabs/databricks with databricks/databricks in all your .tf files.

You should have .terraform.lock.hcl file in your state directory that is checked into source control. terraform init will give you the following warning.

Warning: Additional provider information from registry

The remote registry returned warnings for registry.terraform.io/databrickslabs/databricks:
- For users on Terraform 0.13 or greater, this provider has moved to databricks/databricks. Please update your source in required_providers.

After you replace databrickslabs/databricks with databricks/databricks in the required_providers block, the warning will disappear. Do a global "search and replace" in *.tf files. Alternatively you can run python3 -c "$(curl -Ls https://dbricks.co/updtfns)" from the command-line, that would do all the boring work for you.

If you didn't check-in .terraform.lock.hcl to the source code version control, you may you may see Failed to install provider error. Please follow the simple steps described in the troubleshooting guide.

Use of Terraform exporter

The exporter functionality is experimental and provided as is. It has an evolving interface, which may change or be removed in future versions of the provider.

Well-Known Types

Duration Fields

Duration fields accept Go-style duration strings. Use standard time units like s (seconds), m (minutes), h (hours).

resource "databricks_example" "this" {
  timeout_duration = "30m"   # 30 minutes
  retry_delay      = "10s"   # 10 seconds
  max_lifetime     = "24h"   # 24 hours
  short_timeout    = "500ms" # 500 milliseconds
}

Timestamp Fields

Timestamp fields require RFC3339 formatted datetime strings. Include timezone information for best results.

resource "databricks_example" "this" {
  start_time = "2023-10-15T14:30:00Z"      # UTC timezone
  end_time   = "2023-10-15T14:30:00-07:00" # Pacific timezone
  created_at = "2023-10-15T14:30:00.123Z"  # With milliseconds
}

JSON Value Fields

JSON value fields accept JSON-encoded strings. The provider will normalize and validate the JSON structure.

resource "databricks_example" "this" {
  settings = jsonencode({
    enabled = true
    options = ["option1", "option2"]
  })
}