Using variables in terraform backend config block

[glenjamin]

Terraform Version

v0.9.0

Affected Resource(s)

terraform backend config

Terraform Configuration Files

variable "azure_subscription_id" {
    type = "string"
    default = "74732435-e81f-4a43-bf68-ced435436edf"
}
variable "azure_tenant_id" {
    type = "string"
    default = "74732435-e81f-4a43-bf68-ced435436edf"
}
terraform {
    required_version = ">= 0.9.0"
    backend "azure" {
        resource_group_name = "stuff"
        storage_account_name = "morestuff"
        container_name = "terraform"
        key = "yetmorestuff.terraform.tfstate"
        arm_subscription_id = "${var.azure_subscription_id}"
        arm_tenant_id = "${var.azure_tenant_id}"
    }
}

Expected Behavior

Variables are used to configure the backend

Actual Behavior

Error initializing new backend:
Error configuring the backend "azure": Failed to configure remote backend "azure": Couldn't read access key from storage account: Error retrieving keys for storage account "morestuff": autorest#WithErrorUnlessStatusCode: POST https://login.microsoftonline.com/$%7Bvar.azure_tenant_id%7D/oauth2/token?api-version=1.0 failed with 400 Bad Request: StatusCode=400.

Steps to Reproduce

1. terraform apply

Important Factoids

I wanted to extract these to variables because i'm using the same values in a few places, including in the provider config where they work fine.

[darrensimio]

I am trying to do something like this; getting the same "configuration cannot contain interpolations" error. While it seems like this is being worked on, I wanted to also ask if this is the right way for me to use access and secret keys? Does it have to be placed here so that I don't have to check the access and secret keys to github

terraform {
backend "s3" {
bucket = "ops"
key = "terraform/state/ops-com"
region = "us-east-1"
encrypt = "true"
access_key = "${var.aws_access_key}"
secret_key = "${var.aws_secret_key}"
}
}

[antonosmond]

I have the same problem i.e. would love to see interpolations in the backend config. Now that we have "environments" in terraform, I was hoping to have a single config.tf with the backend configuration and use environments for my states. The problem is that I want to assume an AWS role based on the environment I'm deploying to. I can do this in "provider" blocks as the provider block allows interpolations so I can assume the relevant role for the environment I'm deploying to, however if I also rely on the role being set for the backend state management (e.g. when running terraform env select) it doesn't work. Instead I have to use the role_arn in the backend config which can't contain the interpolation I need.

[darrensimio]

I managed to get it working by using AWS profiles instead of the access keys directly. What I did though was not optimal; but in my build steps, I ran a bash script that called AWS configure that ultimately set the default access key and secret.

[wasfree]

We want to archive something similar than @antonosmond. At the moment we use multiple environments prod/stage and want to upload tfstate files to S3.

## State Backend
terraform {
  backend "s3" {
    bucket  = "mybucket"
    key     = "aws/${var.project}/${var.environment}"
    region  = "eu-central-1"
    profile = "default"
    encrypt = "true"
    lock_table = "terraform"
  }
}

In this case with above backend definition leads us to this Error:

• terraform.backend: configuration cannot contain interpolations

Now if we try to hardcode it like this:

## State Backend
terraform {
  backend "s3" {
    bucket  = "mybucket"
    key     = "aws/example/prod"
    region  = "eu-central-1"
    profile = "default"
    encrypt = "true"
    lock_table = "terraform"
  }
}

we get the following notification:

Do you want to copy only your current environment?
  The existing backend "local" supports environments and you currently are
  using more than one. The target backend "s3" doesn't support environments.
  If you continue, Terraform will offer to copy your current environment
  "prod" to the default environment in the target. Your existing environments
  in the source backend won't be modified. If you want to switch environments,
  back them up, or cancel altogether, answer "no" and Terraform will abort.

Is there a workaround for this problem at the moment, documentation for backend configuration does not cover working with environments.

Solved

seems my local test env was still running on terraform 0.9.1, after updating to latest version 0.9.2 it was working for me.

Do you want to migrate all environments to "s3"?
  Both the existing backend "local" and the target backend "s3" support
  environments. When migrating between backends, Terraform will copy all
  environments (with the same names). THIS WILL OVERWRITE any conflicting
  states in the destination.
  
  Terraform initialization doesn't currently migrate only select environments.
  If you want to migrate a select number of environments, you must manually
  pull and push those states.
  
  If you answer "yes", Terraform will migrate all states. If you answer
  "no", Terraform will abort.

[gsirvas]

Hi,
I'm trying to the the same as @NickMetz, I'm running terraform 0.9.3

$terraform version
Terraform v0.9.3

This is my code
terraform {
  backend "s3" {
    bucket = "tstbckt27" 
    key = "/${var.env}/t1/terraform.tfstate"
    region = "us-east-1"
  }
}

This is the message when I try to run terraform init

$ terraform init
Initializing the backend...
Error loading backend config: 1 error(s) occurred:

* terraform.backend: configuration cannot contain interpolations

The backend configuration is loaded by Terraform extremely early, before
the core of Terraform can be initialized. This is necessary because the backend
dictates the behavior of that core. The core is what handles interpolation
processing. Because of this, interpolations cannot be used in backend
configuration.

If you'd like to parameterize backend configuration, we recommend using
partial configuration with the "-backend-config" flag to "terraform init".

Is this expected behaviour on v0.9.3?

Are there any workarounds for this?

[umeat]

In case it's helpful to anyone, the way I get around this is as follows:

terraform {
  backend "s3" {}
}

data "terraform_remote_state" "state" {
  backend = "s3"
  config {
    bucket     = "${var.tf_state_bucket}"
    lock_table = "${var.tf_state_table}"
    region     = "${var.region}"
    key        = "${var.application}/${var.environment}"
  }
}

All of the relevant variables are exported at the deployment pipeline level for me, so it's easy to init with the correct information for each environment.

terraform init \ 
     -backend-config "bucket=$TF_VAR_tf_state_bucket" \ 
     -backend-config "lock_table=$TF_VAR_tf_state_table" \ 
     -backend-config "region=$TF_VAR_region" \ 
     -backend-config "key=$TF_VAR_application/$TF_VAR_environment"

I don't find this ideal, but at least I can easily switch between environments and create new environments without having to edit any terraform.

[wasfree]

@gsirvas @umeat To archive multiple environment with the same backend configuration it is not necessary to use variables/interpolation .It is expected that is not possible to use variables/interpolation in backend configuration see comment from @christofferh.

Just write it like this:

terraform {
  backend "s3" {
    bucket = "tstbckt27" 
    key = "project/terraform/terraform.tfstate"
    region = "us-east-1"
  }
}

Terraform will split and store environment state files in a path like this:
env:/${var.env}/project/terraform/terraform.tfstate

[umeat]

@NickMetz it's trying to do multiple environments with multiple backend buckets, not a single backend. You can't specify a different backend bucket in terraform environments. In my example you could still use terraform environments to prefix the state file object name, but you get to specify different buckets for the backend.

Perhaps it's better to just give accross account access to the user / role which is being used to deploy your terraform. Deploying your terraform to a different account, but using the same backend bucket. Though it's fairly reasonable to want to store the state of an environment in the same account that it's deployed to.

[wasfree]

@umeat in that case you are right, it is not possible at the moment to use different backends for each environment. It would be more comfortable to have a backend mapping for all environments what is not implemented yet.

[joestump]

Perhaps a middle ground would be to not error out on interpolation when the variable was declared in the environment as TF_VAR_foo? Though this might require making such variables immutable? (Which is fine for my use case; not sure about others.)

[knope]

I also would like to be able to use interpolation in my backend config, using v 0.9.4, confirming this frustrating point still exists. In my use case i need to reuse the same piece of code (without writing a new repo each time i'd want to consume it as a module) to maintain multiple separate statefiles.

[nkhanal0]

Same thing for me. I am using Terraform v0.9.4.

provider "aws" {
	region = "${var.region}"
}

terraform {
	backend "${var.tf_state_backend}" {
		bucket = "${var.tf_state_backend_bucket}"
		key = "${var.tf_state_backend_bucket}/terraform.tfstate"
		region = "${var.s3_location_region}"
	}
}

Here is the error Output of terraform validate:

Error validating: 1 error(s) occurred:

* terraform.backend: configuration cannot contain interpolations

The backend configuration is loaded by Terraform extremely early, before
the core of Terraform can be initialized. This is necessary because the backend
dictates the behavior of that core. The core is what handles interpolation
processing. Because of this, interpolations cannot be used in backend
configuration.

If you'd like to parameterize backend configuration, we recommend using
partial configuration with the "-backend-config" flag to "terraform init".

[kilna-magento]

I needs dis! For many features being developed, we want our devs to spin up their own infrastructure that will persist only for the length of time their feature branch exists... to me, the best way to do that would be to use the name of the branch to create the key for the path used to store the tfstate (we're using amazon infrastructure, so in our case, the s3 bucket like the examples above).

I've knocked up a bash script which will update TF_VAR_git_branch every time a new command is run from an interactive bash session.

This chunk of code would be so beautiful if it worked:

terraform {
  backend "s3" {
    key          = "project-name-${var.git_branch}.tfstate"
    ...
  }
}

Every branch gets its own infrastructure, and you have to switch to master to operate on production. Switching which infrastructure you're operating against could be as easy as checking out a different git branch. Ideally it'd be set up so everything named "project-name-master" would have different permissions that prevented any old dev from applying to it. It would be an infrastructure-as-code dream to get this working.

[kilna-magento]

@NickMetz said...

Terraform will split and store environment state files in a path like this:
env:/${var.env}/project/terraform/terraform.tfstate

Your top-level structure looks nice and tidy for traditional dev/staging/prod ... sure:

env:/prod/project1/terraform/terraform.tfstate
env:/prod/project2/terraform/terraform.tfstate
env:/staging/project1/terraform/terraform.tfstate
env:/staging/project2/terraform/terraform.tfstate
env:/dev/project1/terraform/terraform.tfstate
env:/dev/project2/terraform/terraform.tfstate

But what if you want to stand up a whole environment for project-specific features being developed in parallel? You'll have a top-level key for each story branch, regardless of which project that story branch is in...

env:/prod/project1/terraform/terraform.tfstate
env:/prod/project2/terraform/terraform.tfstate
env:/staging/project1/terraform/terraform.tfstate
env:/staging/project2/terraform/terraform.tfstate
env:/story1/project1/terraform/terraform.tfstate
env:/story2/project2/terraform/terraform.tfstate
env:/story3/project2/terraform/terraform.tfstate
env:/story4/project1/terraform/terraform.tfstate
env:/story5/project1/terraform/terraform.tfstate

It makes for a mess at the top-level of the directory structure, and inconsistency in what you find inside each story-level dir structure. Full control over the paths is ideal, and we can only get that through interpolation.

Ideally I'd want my structure to look like "project/${var.git_branch}/terraform.tfstate", yielding:

project1/master/terraform.tfstate
project1/stage/terraform.tfstate
project1/story1/terraform.tfstate
project1/story4/terraform.tfstate
project1/story5/terraform.tfstate
project2/master/terraform.tfstate
project2/stage/terraform.tfstate
project2/story2/terraform.tfstate
project2/story3/terraform.tfstate

Now, everything you find for a given project is under its directory... so long as the env is hard-coded at the beginning of the remote tfstate path, you lose this flexibility.

Microservices are better versioned and managed discretely per component, rather than dumped into common prod/staging/dev categories which might be less applicable on a per-microservice basis, each one might have a different workflow with different numbers of staging phases leading to production release. In the example above project1 might not even have staging... and project2 might have unit/regression/load-testing/staging phases leading to production release.

[2rs2ts]

you'd think at the very least you'd be allowed to use ${terraform.env}...

[apparentlymart]

In Terraform 0.10 there will be a new setting workspace_key_prefix on the AWS provider to customize the prefix used for separate environments (now called "workspaces"), overriding this env: convention.

[gonzaloserrano]

I know a +1 does not add much but yeah, need this too to have 2 different buckets, since we have 2 AWS accounts.

[mhowell-ims]

I was hoping to do the same thing as described in #13603 but the lack of interpolation in the terraform block prevents this.

[Heiko-san]

+1

[Shocktrooper]

@rootsher With terragrunt just switch the backend to using a generate block and not the terragrunt native backend block. We do interpolation that way which works just fine.

[brsolomon-deloitte]

We were able to get around this by using backend-config when initializing the Terraform project as shown below.

terraform {
  backend "s3" {
    profile = "default"
    encrypt = "true"
    lock_table = "terraform"
  }
}

$ terraform init \
     -backend-config="bucket=mybucket" \
     -backend-config="key=mykey" \
     -backend-config="region=us-east-1"

Reference : https://www.terraform.io/language/settings/backends/configuration

Sure, this "works", but it is completely against the very purpose of Terraform, which is to declaratively store a complete picture of resources as code. Adding required parameters from the command line, in the absence of being able to actually using variables within backend, is simply suboptimal. Just as suboptimal as augmenting Terraform with shell scripts or any other solution besides the Terraform developers fixing an issue that's now been open for over 5 years.

[tranvanthuc]

I tried use flag backend-config and it works.
You can check this example
terraform init -backend-config=backend.hcl

[dimisjim]

@tranvanthuc per tf docs: https://developer.hashicorp.com/terraform/language/settings/backends/configuration

thus what is the advantage of using -backend-config cli argument?

[tranvanthuc]

@dimisjim I just need dynamic backend block inside terraform block. Because in these blocks, we can't use variables

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
  backend "s3" {}
}

backend.hcl for staging environment

bucket         = "app-state-staging"
key               = "terraform.tfstate"
region          = "ap-southeast-1"
dynamodb_table = "app-state-staging-locking"

backend.hcl for prod environment

bucket         = "app-state-prod"
key               = "terraform.tfstate"
region          = "ap-southeast-1"
dynamodb_table = "app-state-prod-locking"

Use case:

• Dynamic config in backend "s3" {} (can't use variable inside terraform block)

[dimisjim]

Sure, but the .hcl file is gonna be the variable instead? Wouldn't you need to anyway commit this as well to git? Or are you planning to somehow not track this file in git and/or mask it with env vars somehow?

[tranvanthuc]

Yes, I won't track this file in git. May you have any suggestion for this use case?

[lllamnyp]

I came up with a commit that let's me use all functions and the path.{cwd,root,module} attributes in the backend configuration. It can be taken further to include variables and be made somewhat neater, however, I don't have the time, nor the knowledge of the codebase to turn this into a complete PR. If anyone would like to take this further, or just grab it for their own use, they're welcome.

diff --git a/internal/command/meta_backend.go b/internal/command/meta_backend.go
index 908e87b08..60acd5ce0 100644
--- a/internal/command/meta_backend.go
+++ b/internal/command/meta_backend.go
@@ -19,6 +19,7 @@ import (
 
 	"github.com/hashicorp/hcl/v2"
 	"github.com/hashicorp/hcl/v2/hcldec"
+	"github.com/hashicorp/terraform/internal/addrs"
 	"github.com/hashicorp/terraform/internal/backend"
 	"github.com/hashicorp/terraform/internal/cloud"
 	"github.com/hashicorp/terraform/internal/command/arguments"
@@ -1329,7 +1330,16 @@ func (m *Meta) backendConfigNeedsMigration(c *configs.Backend, s *legacy.Backend
 
 	schema := b.ConfigSchema()
 	decSpec := schema.NoneRequired().DecoderSpec()
-	givenVal, diags := hcldec.Decode(c.Config, decSpec, nil)
+
+	terraform.DefaultEvaluator.Meta.OriginalWorkingDir = m.WorkingDir.OriginalWorkingDir()
+	scope := terraform.DefaultEvaluationStateData.Evaluator.Scope(terraform.DefaultEvaluationStateData, nil, nil)
+	ctx, _ := scope.EvalContext([]*addrs.Reference{
+		{Subject: addrs.PathAttr{Name: "cwd"}},
+		{Subject: addrs.PathAttr{Name: "root"}},
+		{Subject: addrs.PathAttr{Name: "module"}},
+	})
+
+	givenVal, diags := hcldec.Decode(c.Config, decSpec, ctx)
 	if diags.HasErrors() {
 		log.Printf("[TRACE] backendConfigNeedsMigration: failed to decode given config; migration codepath must handle problem: %s", diags.Error())
 		return true // let the migration codepath deal with these errors
@@ -1366,7 +1376,16 @@ func (m *Meta) backendInitFromConfig(c *configs.Backend) (backend.Backend, cty.V
 
 	schema := b.ConfigSchema()
 	decSpec := schema.NoneRequired().DecoderSpec()
-	configVal, hclDiags := hcldec.Decode(c.Config, decSpec, nil)
+
+	terraform.DefaultEvaluator.Meta.OriginalWorkingDir = m.WorkingDir.OriginalWorkingDir()
+	scope := terraform.DefaultEvaluationStateData.Evaluator.Scope(terraform.DefaultEvaluationStateData, nil, nil)
+	ctx, _ := scope.EvalContext([]*addrs.Reference{
+		{Subject: addrs.PathAttr{Name: "cwd"}},
+		{Subject: addrs.PathAttr{Name: "root"}},
+		{Subject: addrs.PathAttr{Name: "module"}},
+	})
+
+	configVal, hclDiags := hcldec.Decode(c.Config, decSpec, ctx)
 	diags = diags.Append(hclDiags)
 	if hclDiags.HasErrors() {
 		return nil, cty.NilVal, diags
diff --git a/internal/terraform/eval_default.go b/internal/terraform/eval_default.go
new file mode 100644
index 000000000..a315234a0
--- /dev/null
+++ b/internal/terraform/eval_default.go
@@ -0,0 +1,14 @@
+package terraform
+
+import "github.com/hashicorp/terraform/internal/configs"
+
+var (
+	DefaultEvaluator           = new(Evaluator)
+	DefaultEvaluationStateData = new(evaluationStateData)
+)
+
+func init() {
+	DefaultEvaluator.Meta = &ContextMeta{Env: "default"}
+	DefaultEvaluator.Config = &configs.Config{Module: &configs.Module{SourceDir: "."}}
+	DefaultEvaluationStateData.Evaluator = DefaultEvaluator
+}

[ZaxLofful]

Is there any plan to implement this soon?

I want to use variables that are set to be used as my access_key, to help with secrets concealment.

[thalesfsp]

@ZaxLofful -> https://github.com/opentffoundation/opentf

[dimisjim]

@thalesfsp which means?

[Theaxiom]

@dimisjim It means that Terraform is now the MySQL to MariaDB and it appears OpenTofu is MariaDB.

[jan-di]

@gothrek22 This issue is about the backend configuration block, not provider configurations.
https://developer.hashicorp.com/terraform/language/settings/backends/configuration

[gothrek22]

@jan-di you are completely right, I've answered in the wrong issue. Just deleted

[vt-rcheng]

I don't think this will ever be fixed cuz they want you to pay for terraform cloud

[Temikus]

Work in the fork for those interested: opentofu/opentofu#1042

[aprilmintacpineda]

An alternative is to use partial backends https://developer.hashicorp.com/terraform/language/settings/backends/configuration#partial-configuration

tl;dr;

1. Omit parts of your backend config that you want to be variables

terraform {
  backend "s3" {
    bucket = "my-terraform-backends"
    region = "ap-southeast-1"
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.53"
    }
  }
}

In above, I omitted the key.

2. Create a file called config.s3.tfbackend

You'd want to treat this file similar to how you treat your terraform.tfvars file, here, you'll put all the variables for your backend but using the exact same property key names. Below, I specified a key which corresponds to my terraform.backend.key configuration.

key = "myproject/development"

3. Run terraform init to reinitialize your backend

Run terraform init -backend-config=config.s3.tfbackend. If you're running in CI, you'd probably need to call this every single run before running apply. Now terraform will start using your s3 bucket as its backend.

[thalesfsp]

@dimisjim commented [on Sep 10, 2023]
@thalesfsp which means?

@Theaxiom commented [on Oct 3, 2023]
@dimisjim It means that Terraform is now the MySQL to MariaDB and it appears OpenTofu is MariaDB.

@Theaxiom awesome explanation. Thank you for translating.

@dimisjim Also... this issue is open since 2017, lol.

[lijok]

This is now supported in OpenTofu: https://github.com/opentofu/opentofu/releases/tag/v1.8.0-alpha1

[akashbogamibm]

what are the possible way to use variable in terraform backend, I'm asking this because I'm storing my code in GitHub repo, I cannot expose credentials there. can anyone help me with this ?

[magic-happenz]

@akashbogamibm why can't? GitHub got secret variables for exactly such purpose. You don't have to "expose" anything.