Working with Infrastructure as Code (IaC) is incredible. Figuring out the correct repository structure, including a CI/CD pipeline, is not so simple.
Here are my two cents on the subject.
This can be done manually in the AWS Console or by using a terraform module, such as cloudposse/terraform-aws-tfstate-backend.
Initially, I used the terraform module, but then I found out I'm stuck in the "chicken and the egg" situation. I'm using terraform to create resources that will save the state of those resources; what?
I found it easier to do this task with CloudFormation, which makes it totally a separate process, and it is done once per environment.
I've created the script scripts/prepare-backend.sh, which creates an S3 bucket for the state file and DynamoDB table for state lock. The script uses aws-cli and deploys the cloudformation/cfn-tfbackend.yml template.
NOTE: Remote Backend resources are created per environment (branch).
One of the major pain points of this configuration is setting the Terraform Remote Backend per environment. Each environment must have its own "backend settings", and the values must be hard-coded since you can't use variables in the terraform backend code block.
Only one backend may be specified, and the configuration may not contain interpolations. Terraform will validate this. Source
This part was solved by creating a template, which gets modified before executing terraform apply.
The script that applies the modification is scripts/prepare-files-folders.sh. It's using a simple sed command to find and replace AWS_REGION, APP_NAME and ENVIRONMENT. After applying these changes, the file backend.tf.tpl is renamed to backend.tf, making it available to the terraform CLI due to its tf extension.
This script prepare-files-folders.sh has a bigger purpose, and we'll dive into it in a few moments.
The goal is to have a single directory that contains all of the IaC files (tf, JSON, and so on) and then execute terraform apply from this directory. And again, scripts/prepare-files-folders.sh comes to the rescue.
A new directory dev was created in the project's root directory, and it contains the files that are needed to deploy the infrastructure. The example above is for dev, but the result would be the same for stg and prd; the only thing that really changes is the Remote Backend configuration.
The variable BRANCH_NAME is assigned to environment, according to the branch that will be deployed with terraform apply. The CI/CD process sets the value of BRANCH_NAME (we'll get to that).
NOTE: I prefer using local values in my resources and modules instead of variables. I even wrote a blog post about it. It's not mandatory, but I consider it as best practice.
This part is fully covered in the README of terraform-multienv. This is an abstract of the whole process.
You might have noticed I didn't mention anything about scripts/terraform.sh. Since you've got this far, you deserve to know what it does
I hope you find this useful, and if you do, don't forget to clap/heart and share it with your friends and colleagues.
Got any questions or doubts? Let's start a discussion! Feel free to comment below.