At this point, I spend a large part of my week inside of the Amazon Web Services ecosystem. If I had to make a guess I would say 85% of the day is creating, updating, or destroying AWS infrastructure.
But, I spend less than 1% of my week inside of the AWS Console.
That is to say that I don’t touch any AWS infrastructure via the console. All the infrastructure I interact with is in code, Terraform HCL to be specific.
When infrastructure is in a declarative language like HCL, provisioning, updating, and deleting is easier, less error-prone, and reproducible. It is also transparent. Every change to the infrastructure goes through a pull request process that is reviewed by peers.
For all the positives with infrastructure as code, particularly HCL, there are downsides and negatives.
- HCL is yet another language to learn. As we will see in a minute, it’s not like the language you are likely creating your application in.
- It’s limited in the set of features it offers.
- Modularity can be difficult to get started with if you’re not familiar with it.
Point (3) is the one we are going to focus on here. With Terraform modules we can create infrastructure packages that are reusable.
Working from an example
Let’s take a basic example, creating an IAM user with the necessary policy to access Amazon Elastic Container Registry (ECR). Here are the three resources we need in Terraform.
# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
name = "some-user"
}
resource "aws_iam_user_policy" "some-user" {
name = "some-user"
user = aws_iam_user.some-user.name
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = ["ecr:GetAuthorizationToken"],
Resource = "*"
},
{
Effect = "Allow",
Action = [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
Resource = "*"
}
]
})
}
resource "aws_iam_access_key" "some-user" {
user = aws_iam_user.some-user.name
}
To ground ourselves in reality, let’s put these into a real Terraform project. Run the following commands in the Terminal. (note: you will need to install Terraform and have the aws cli installed).
$ mkdir terraform-example
$ cd terraform-example
$ touch main.tf
Now let’s add the following block to main.tf.
terraform {
required_version = ">=0.12, <0.13"
}
provider "aws" {
version = "~> 2.58"
region = "us-west-2"
}
This requires that we have a terraform version greater than or equal to 0.12, but less than 0.13. The second bit is
where we are declaring that we are going to use the AWS provider. This is the provider that will allow us to access AWS
resources inside of our account.
Below the provider, let’s add in the IAM resources from earlier. We should now have something like this in main.tf.
terraform {
required_version = ">=0.12, <0.13"
}
provider "aws" {
version = "~> 2.58"
region = "us-west-2"
}
# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
name = "some-user"
}
resource "aws_iam_user_policy" "some-user" {
name = "some-user"
user = aws_iam_user.some-user.name
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = ["ecr:GetAuthorizationToken"],
Resource = "*"
},
{
Effect = "Allow",
Action = [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
Resource = "*"
}
]
})
}
resource "aws_iam_access_key" "some-user" {
user = aws_iam_user.some-user.name
}
Let’s run this template and confirm that a new IAM user gets created. From the terraform-example folder run the
following commands.
$ terraform init
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_iam_access_key.some-user will be created
+ resource "aws_iam_access_key" "some-user" {
+ encrypted_secret = (known after apply)
+ id = (known after apply)
+ key_fingerprint = (known after apply)
+ secret = (sensitive value)
+ ses_smtp_password = (sensitive value)
+ ses_smtp_password_v4 = (sensitive value)
+ status = (known after apply)
+ user = "some-user"
}
# aws_iam_user.some-user will be created
+ resource "aws_iam_user" "some-user" {
+ arn = (known after apply)
+ force_destroy = false
+ id = (known after apply)
+ name = "some-user"
+ path = "/"
+ unique_id = (known after apply)
}
# aws_iam_user_policy.some-user will be created
+ resource "aws_iam_user_policy" "some-user" {
+ id = (known after apply)
+ name = "some-user"
+ policy = jsonencode(
{
+ Statement = [
+ {
+ Action = [
+ "ecr:GetAuthorizationToken",
]
+ Effect = "Allow"
+ Resource = "*"
},
+ {
+ Action = [
+ "ecr:GetAuthorizationToken",
+ "ecr:BatchCheckLayerAvailability",
+ "ecr:GetDownloadUrlForLayer",
+ "ecr:GetRepositoryPolicy",
+ "ecr:DescribeRepositories",
+ "ecr:ListImages",
+ "ecr:DescribeImages",
+ "ecr:BatchGetImage",
+ "ecr:InitiateLayerUpload",
+ "ecr:UploadLayerPart",
+ "ecr:CompleteLayerUpload",
+ "ecr:PutImage",
]
+ Effect = "Allow"
+ Resource = "*"
},
]
+ Version = "2012-10-17"
}
)
+ user = "some-user"
}
Plan: 3 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:
The apply step has stopped us to confirm these are all the resources we want to create. Go ahead and type yes in the
prompt. We should see the following after the apply.
aws_iam_user.some-user: Creating...
aws_iam_user.some-user: Creation complete after 1s [id=some-user]
aws_iam_user_policy.some-user: Creating...
aws_iam_access_key.some-user: Creating...
aws_iam_access_key.some-user: Creation complete after 0s [id=<some-id>]
aws_iam_user_policy.some-user: Creation complete after 1s [id=some-user:some-user]
Awesome, we now have an IAM user, some-user, that has a policy that allows them to access ECR. If we only ever had to
create one user, were set, nothing more to do here.
But, we often need more than one user. It also tends to work out that we need to give each user the same permissions.
Using modules
We could have more than one IAM user with the same policy by copying and pasting the resources. So if we had a second
user, our main.tf could look like this.
terraform {
required_version = ">=0.12, <0.13"
}
provider "aws" {
version = "~> 2.58"
region = "us-west-2"
}
# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
name = "some-user"
}
resource "aws_iam_user_policy" "some-user" {
name = "some-user"
user = aws_iam_user.some-user.name
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = ["ecr:GetAuthorizationToken"],
Resource = "*"
},
{
Effect = "Allow",
Action = [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
Resource = "*"
}
]
})
}
resource "aws_iam_access_key" "some-user" {
user = aws_iam_user.some-user.name
}
# Second user
resource "aws_iam_user" "another-user" {
name = "another-user"
}
resource "aws_iam_user_policy" "another-user" {
name = "another-user"
user = aws_iam_user.another-user.name
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = ["ecr:GetAuthorizationToken"],
Resource = "*"
},
{
Effect = "Allow",
Action = [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
Resource = "*"
}
]
})
}
resource "aws_iam_access_key" "another-user" {
user = aws_iam_user.another-user.name
}
We can look at the another-user resources and see that we just copied and pasted the resources. This works and is OK
in a pinch. But imagine if you had to do this for 10 users 😭
This is where Terraform modules come in to save the day. A module is a collection of resources that belong together. They can be called from other modules which adds the resources from the module.
In fact, we are already using a module in our example. We are using the root module. The root module is always the
entry main.tf defined in our project.
We are going to convert the three resources, IAM user, IAM policy, and IAM access key into a module that can be reused.
To get started, create a new folder at the root of terraform-example for our module to live in.
$ mkdir modules
$ cd modules
$ touch iam.tf
Now inside of iam.tf let’s add our three resources from earlier.
# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
name = "some-user"
}
resource "aws_iam_user_policy" "some-user" {
name = "some-user"
user = aws_iam_user.some-user.name
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = ["ecr:GetAuthorizationToken"],
Resource = "*"
},
{
Effect = "Allow",
Action = [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
Resource = "*"
}
]
})
}
resource "aws_iam_access_key" "some-user" {
user = aws_iam_user.some-user.name
}
Right now this IAM module is not parameterized. Let’s go ahead and fix that by adding a username variable at the
top iam.tf.
# IAM User + Policy + Key to access ECR
variable "username" {
type = string
}
resource "aws_iam_user" "some-user" {
name = var.username
}
resource "aws_iam_user_policy" "some-user" {
name = "some-user"
user = aws_iam_user.some-user.name
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = ["ecr:GetAuthorizationToken"],
Resource = "*"
},
{
Effect = "Allow",
Action = [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
Resource = "*"
}
]
})
}
resource "aws_iam_access_key" "some-user" {
user = aws_iam_user.some-user.name
}
Now our entire IAM module takes in a username variable. It uses that to create the IAM user, IAM policy, and access
key for that username.
Reusing modules
Now that we have an IAM module that takes in a username we can go back and edit our main.tf. We no longer have to
copy and paste resources to add a new user. Instead, we can call the module with the username we want to create.
Here is an example where we are creating three users.
terraform {
required_version = ">=0.12, <0.13"
}
provider "aws" {
version = "~> 2.58"
region = "us-west-2"
}
module "user-1" {
source = "./modules"
username = "some-user-2"
}
module "user-2" {
source = "./modules"
username = "another-user"
}
module "user-3" {
source = "./modules"
username = "yet-another-user"
}
Instead of copying and pasting resources around, we use the module we created. This module has all the resources that
need to be created. From the outside, we give the module the username we want the IAM user to have.
Let’s do a plan on this and see what happens. We have to run terraform init again because added a new module.
$ terraform init
$ terraform plan
We see that nine resources are going to be created which makes sense because we are creating three new users. If we look at the plan we see that resources are now coming from the module.
# module.user-3.aws_iam_access_key.some-user will be created
+ resource "aws_iam_access_key" "some-user" {
+ encrypted_secret = (known after apply)
+ id = (known after apply)
+ key_fingerprint = (known after apply)
+ secret = (sensitive value)
+ ses_smtp_password = (sensitive value)
+ ses_smtp_password_v4 = (sensitive value)
+ status = (known after apply)
+ user = "yet-another-user"
}
# module.user-3.aws_iam_user.some-user will be created
+ resource "aws_iam_user" "some-user" {
+ arn = (known after apply)
+ force_destroy = false
+ id = (known after apply)
+ name = "yet-another-user"
+ path = "/"
+ unique_id = (known after apply)
}
# module.user-3.aws_iam_user_policy.some-user will be created
+ resource "aws_iam_user_policy" "some-user" {
+ id = (known after apply)
+ name = "some-user"
+ policy = jsonencode(
{
+ Statement = [
+ {
+ Action = [
+ "ecr:GetAuthorizationToken",
]
+ Effect = "Allow"
+ Resource = "*"
},
+ {
+ Action = [
+ "ecr:GetAuthorizationToken",
+ "ecr:BatchCheckLayerAvailability",
+ "ecr:GetDownloadUrlForLayer",
+ "ecr:GetRepositoryPolicy",
+ "ecr:DescribeRepositories",
+ "ecr:ListImages",
+ "ecr:DescribeImages",
+ "ecr:BatchGetImage",
+ "ecr:InitiateLayerUpload",
+ "ecr:UploadLayerPart",
+ "ecr:CompleteLayerUpload",
+ "ecr:PutImage",
]
+ Effect = "Allow"
+ Resource = "*"
},
]
+ Version = "2012-10-17"
}
)
+ user = "yet-another-user"
}
Now if we go and apply this we should see that three new users get created. (note: I use the -auto-approve flag to
tell Terraform to auto-approve the apply without prompting me).
$ terraform apply -auto-approve
module.user-1.aws_iam_user_policy.some-user: Creation complete after 0s [id=some-user-2:some-user]
module.user-1.aws_iam_access_key.some-user: Creation complete after 0s [id=<some-id>]
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
Now we see that nine resources got added. That is because each module for our IAM user has three resources, the user, the IAM policy, and the IAM access key.
With that, we now have an IAM user Terraform module that we can reuse to create new IAM users. But it’s actually even better than that, we can also leverage this to update users.
Because each user is created via the module, any change we make to the module also gets applied to every user. This is handy, for example, if we wanted to have a common IAM policy that each user had associated with them.
Conclusion
Infrastructure as code is a powerful tool within a cloud environment. It allows your infrastructure to live next to your everyday code. It encourages knowledge sharing and pull request processes on infrastructure changes. With the right tools, it allows you to share common infrastructure ideas across teams.
What we saw here was one example of a tool, Terraform. We took a look at how we can use Terraform modules to create packages of infrastructure that are reusable.
In this example, we focused on something straight forward, an IAM user. But modules can be created for any number of things. Perhaps you want to create a module that represents every piece of infrastructure for your application. Or maybe you want each team to use a common VPC module that has security at the forefront. Whatever the case, modules allow you to create reusable blocks of resources.