Build infrastructure CI for Terraform code Using AWS Dev Tools & Terratest

Customers are increasingly using Infrastructure as Code (IaC) as the primary method for designing, providing, developing, and maintaining their cloud infrastructure. IaC ensures the infrastructure’s repeatability and scalability while also allowing DevOps to embrace best practices and concepts. Terraform is frequently used by customers to supply and manage infrastructure and services on AWS in the context of IaC. Developers and infrastructure architects can use Terraform to organise their code into reusable and shareable modules. Modules may and should be used to construct lightweight abstractions so that your infrastructure can be described in terms of its design rather than physical things.

As a result, running automated tests on the IaC becomes a critical task. These automated tests guarantee that the code, and thus the given resources, are of high quality, stability, reliability, and security. Setting up automated test pipelines is difficult due to the particular nature of Infrastructure as Code, which forces developers to use non-standard self-customized solutions.

Solution Overview

This blog post’s goal is to give step-by-step instruction and ready-to-use code for setting up a complete Continuous Integration (CI) pipeline with end-to-end tests using AWS CodePipeline, AWS CodeBuild, AWS CodeCommit, Terraform, and Terratest.

As a result, users will be able to use the entire CI Pipeline as well as modify and customise the code to meet their specific requirements. The proposed solution is depicted in the architecture diagram below. It’s worth noting that the end-to-end tests based on the given Terraform modules can be conducted using an independent account.

Repository Structure

Please download the contents of the following GitHub repository as a starting point.
A description of the code’s structure may be found below. It will assist you in navigating through the following example:

Bash
.
|-- README.md
`-- terraform
|-- examples
`-- modules
/terraform

This directory contains:
./examples reusable examples for the modules.
./modules The CodeBuild build scripts are included in the Terraform modules for the CI pipeline.
The repository’s material is organised to provide reusable Terraform modules.
These modules, as well as the content they contain, can be extracted and reused.

/terraform/examples

When publishing modules, it’s a good idea to include some functioning examples. Those examples can be kept in a separate directory underneath the modules directory.

In this repository the example cicd_account (on the location: /terraform/examples/cicd_account/main.tf) is provided. It uses the module located in /terraform/modules/cicd.  This example will deliver a CI based on CodePipeline, CodeBuild and CodeCommit and uses this same repository as the source.

It should be noted that the proposed behaviour can be modified and tailored to meet the demands of the individual.

/terraform/modules

The Terraform modules, tests, and CodeBuild code are all located in this directory.

For more information on CodeBuild and how to understand the buildspec.yaml file, go to Build specification documentation for CodeBuild.

In a nutshell, the CodeBuild workflow is as follows:

1. It runs the tests described in modules/cicd/test
2. The test report for each run can be seen in the Reports section of the AWS CodeBuild Console.

Prerequisites

For this walkthrough, you should have the following prerequisites:

• An AWS Account 
• Terraform version ≥ 12.0
• aws CLI v2
• python ≥3.8
• go ≥1.15.2
• Terratest

Walkthrough

1. Deploy CI Pipeline

The CI pipeline can be deployed using the provided code example. The cross-account arrangement, in which the CI runs on one AWS account and the tests are conducted on another AWS account, is an option. This post will walk you through setting up the solution in a single AWS account for the purpose of simplicity.

To deploy the CI pipeline follow these steps:

1. Modify the file under:
   terraform/examples/cicd_account/inventories/variables.tfvars with the desired name of your repository:
   git_repository_name=”artifact-cicd-test”
2. Configure your AWS CLI
3. Deploy the resources with and follow the instructions provided:
   cd terraform/examples
   ./run_terraform.sh cicd_account

This is what you’ve set up in your account —

• AWS CodeCommit — AWS CodeCommit is a fully-managed source control solution that maintains a secure Git repository.
• AWS CodeBuild — I used this service to conduct continuous integration, which is why the terraform tests are there.
• AWS CodePipeline – AWS CodePipeline is a fully managed continuous delivery solution that automates the build/test process.
• The least-privilege permissions required for the scope of this example are described in AWS IAM Roles and Policies.

2. Review and tune the test code

A test example called  cicd_test.go is provided in the location terraform/modules/cicd/test
The example can be extended to meet the desired or required requirements. This step-by-step guide to the module cicd can also be found in the terraform/modules/cicd/README

3. Push the code inside your new CodeCommit repository.

You can use the same code you used in Step 1 for this sample test as if it were the code you want to test. To push the Terraform code to the newly formed CodeCommit repository, follow these steps:

1. This first step is optional, but it is recommended: to create a .gitignore file in the root level of the repository to avoid git tracking Terraform state:

Bash
echo “.tfstate*
.terraform
plan.out” >> .gitignore

2. Retrieve repository remote address:

• Go to CodeCommit in the AWS console.
• Select your repository by clicking on its name.
• Click Clone URL and choose your chosen method (HTTPS (GRC) is suggested).
• Clicking on connecting steps will take you to more detailed documentation on how to set it up.

3. Setup your git repo:

Bash
git remote add origin <THE_URL_YOU_GOT_ON_STEP2>
## By default the CodePipeline monitors the dev branch
git checkout -b dev
git push –set-upstream origin

4. Push the repo against the dev branch this will trigger the start of the Pipeline
5. Follow the pipeline run by connecting to the CodePipeline console.

4. Cleanup

When finished you can easily clean up your account by following the next steps:

Bash
cd terraform/examples
./run_terraform.sh cicd_account delete

Troubleshooting

Terraform Error when deleting S3 Bucket (BucketNotEmpty)

When you’re breaking down or changing the infrastructure, Terraform may quit with this error. The reason for this is that Terraform will not remove an S3 bucket that is not empty. To resolve the problem, manually remove the following object from the S3 Bucket:

1. Log in to the AWS Console and navigate to the S3 bucket.
2. Select the bucket you want to empty and click Empty.
3. On the console, confirm the deletion.
4. Once the operation is complete, try running the terraform command again; it should now work.

As an alternative, you may also make use of  force_destroy argument in the aws_s3_bucket Terraform resource. Follow this link for more information

Conclusion

Customers may now set up a fully automated CI pipeline for testing Terraform code using Terratest libraries, AWS CodePipeline, AWS CodeBuild, and AWS CodeCommit after following this guide. Customers are now familiar with the testing principle for IaC, specifically for Terraform, as a result of this.

Related/References

• Terraform Installation Overview
• Variables in Terraform
• Terraform Providers Overview
• Everything you need to know about Terraform Certification

Next Task For You

To know more about Terraform and get access to all Hands-On labs that you must perform to clear the Terraform Certified Associate certification exam, Register for our FREE Masterclass.

The post Build infrastructure CI for Terraform code Using AWS Dev Tools & Terratest appeared first on Cloud Training Program.