Tracking AWS cost continues to be a very popular topic in the world of the cloud. The reasoning is often because folks are getting surprised by their bills at the end of the month. Or perhaps they thought a service was going to cost X, but at the end of the month it ended up costing 2X.
This isn’t just a problem for those of us that are new to AWS. This is an issue that even the most seasoned AWS developers are running into.
Take me for an example, this morning I woke up to an AWS bill that was $20 more than what I typically pay each month. I have a billing alarm set up for my entire account, but the total bill still came in under that so it wasn’t breached.
So I got to digging around and discovered that the additional spend was because I left a Load Balancer running in my account that I was using for testing.
This scenario is very common for folks because it is very easy to forget about a particular resource you created. Sometimes this is because we created the resource manually in the AWS Console. Or we created some infrastructure via code, but then neglected to tear it down when we were done with it.
For one of my products, the monthly cost for my RDS and EC2 spend is very stable and predictable. But in this case, because I left a Load Balancer running, my EC2 cost was $20 more than usual. So I set out to fix the issue of not knowing I left a Load Balancer running.
The goal of this post is to demonstrate how we can leverage AWS Budgets to create per service level billing alarms. We are going to use Terraform to create our budgets so make sure you have that installed and in your path.
What we want to track
Ultimate Fantasy Supercross is a fantasy motorsports platform I launched about three years ago. It uses a couple EC2 instances, an RDS database, and a static website in S3. The cost of my infrastructure is very predictable at this point.
| Service | Monthly Cost ($) |
|---|---|
| Elastic Compute Cloud (EC2) | 19.99 |
| Relational Database Service (RDS) | 35.04 |
| Simple Storage Service (S3) | 1.50 |
As mentioned earlier I already have an account level billing alarm configured that notifies me when my bill is approaching the $60 mark. But I also want to know when my forecasted spend is going to breach my monthly expectation for EC2 and RDS.
To accomplish this we are going to create two AWS Budgets, one for EC2 and one for RDS, to send me an email when my forecasted spend for each of these is going to exceed my expectation.
Creating our budgets
We are going to create three resources in our Terraform template. The first will be an account level billing alarm just in case you don’t already have one. Then we will create two budget resources for our EC2 and RDS forecasted spend.
Let’s begin with our account level billing alarm.
provider "aws" {
region = "us-east-1"
}
resource "aws_cloudwatch_metric_alarm" "account-billing-alarm" {
alarm_name = "account-billing-alarm"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = "1"
metric_name = "EstimatedCharges"
namespace = "AWS/Billing"
period = "21600"
statistic = "Average"
threshold = "60"
alarm_description = "Billing alarm by account"
alarm_actions = ["<your-sns-topic-arn-for-notification>"]
dimensions {
Currency = "USD"
LinkedAccount = "<your-aws-account-id>"
}
}Here we have our account level billing alarm. It is tracking the EstimatedCharges for an individual account. When that
account reaches $60 USD in estimated charges it will send an alarm notification to the SNS topic we specify in
alarm_actions. You should replace alarm_actions with the ARN of an SNS topic that has your email subscribed to
it. You also need to replace LinkedAccount with your own AWS account id.
We can create our account level alarm by first initializing our template.
$ terraform init
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.39.0)...Once our template is initialized we can apply our template to our AWS account.
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_cloudwatch_metric_alarm.account-billing-alarm
....
....
aws_cloudwatch_metric_alarm.account-billing-alarm: Creating...
actions_enabled: "" => "true"
alarm_actions.#: "" => "1"
alarm_actions.321893454: "" => "<your-sns-topic-arn-for-notification>"
alarm_description: "" => "Billing alarm by account"
alarm_name: "" => "account-billing-alarm"
arn: "" => "<computed>"
comparison_operator: "" => "GreaterThanOrEqualToThreshold"
dimensions.%: "" => "2"
dimensions.Currency: "" => "USD"
dimensions.LinkedAccount: "" => "<your-aws-account-id>"
evaluate_low_sample_count_percentiles: "" => "<computed>"
evaluation_periods: "" => "1"
metric_name: "" => "EstimatedCharges"
namespace: "" => "AWS/Billing"
period: "" => "21600"
statistic: "" => "Average"
threshold: "" => "60"
treat_missing_data: "" => "missing"
aws_cloudwatch_metric_alarm.account-billing-alarm: Creation complete after 1s (ID: account-billing-alarm)
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Great! We now have our account level billing alarm to notify us when our estimated bill is going to be greater than $60 USD. Let’s create our budgets so we can alarm when our forecasted spend on EC2 or RDS exceeds our expectation.
Go ahead and add the following resources to your Terraform template.
resource "aws_budgets_budget" "ec2-forecast-alarm" {
name = "budget-ec2-monthly"
budget_type = "COST"
limit_amount = "20"
limit_unit = "USD"
time_period_start = "2018-01-01_00:00"
time_unit = "MONTHLY"
cost_filters {
service = "Amazon Elastic Compute Cloud - Compute"
}
}
resource "aws_budgets_budget" "rds-forecast-alarm" {
name = "budget-rds-monthly"
budget_type = "COST"
limit_amount = "36"
limit_unit = "USD"
time_period_start = "2018-01-01_00:00"
time_unit = "MONTHLY"
cost_filters {
service = "Amazon Relational Database Service"
}
}Here we have our budgets for EC2 and RDS. Notice the cost_filters are using the fully qualified names for each
service, this is what AWS budgets via Terraform expect. We can go ahead and run our apply command again to create our
new budgets.
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_budgets_budget.ec2-forecast-alarm
id: <computed>
account_id: <computed>
budget_type: "COST"
cost_filters.%: "1"
cost_filters.Service: "ec2"
cost_types.#: <computed>
limit_amount: "20"
limit_unit: "USD"
name: "budget-ec2-monthly"
name_prefix: <computed>
time_period_end: "2087-06-15_00:00"
time_period_start: "2018-01-01_00:00"
time_unit: "MONTHLY"
+ aws_budgets_budget.rds-forecast-alarm
id: <computed>
account_id: <computed>
budget_type: "COST"
cost_filters.%: "1"
cost_filters.Service: "rds"
cost_types.#: <computed>
limit_amount: "36"
limit_unit: "USD"
name: "budget-rds-monthly"
name_prefix: <computed>
time_period_end: "2087-06-15_00:00"
time_period_start: "2018-01-01_00:00"
time_unit: "MONTHLY"
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.Just like that we now have two budgets configured in our AWS account. The first tracks our spend on EC2 and the latter
tracks our spend on RDS. However, we do not have notifications set up for these yet. This is an
open issue in Terraform to add this support
to the aws_budgets resource, but at this point, we have to do this ourselves.
Adding notifications to our budgets
Because of the limitation of Terraform as of today, we need to add the notifications to our budgets outside of our template. We can quickly add notifications to each of our budgets by using the AWS CLI. To get started you need your AWS account id once more.
Got it? Great, now we can run the following commands on our command line.
$ aws budgets create-notification --account-id <your-aws-account-id> --budget-name budget-rds-monthly --notification NotificationType=FORECASTED,ComparisonOperator=GREATER_THAN,Threshold=100,ThresholdType=PERCENTAGE --subscribers SubscriptionType=EMAIL,Address=<your-email-address>
$ aws budgets create-notification --account-id <your-aws-account-id> --budget-name budget-ec2-monthly --notification NotificationType=FORECASTED,ComparisonOperator=GREATER_THAN,Threshold=100,ThresholdType=PERCENTAGE --subscribers SubscriptionType=EMAIL,Address=<your-email-address>Voila! We now have two AWS budgets configured in our account to notify us when our spend for either EC2 or RDS breaches the amount we expect.
Conclusion
Getting surprised by the cost of your AWS resources is not a great feeling. It’s even less of a good feeling when the cost is unnecessary due to something like leaving a load balancer running. Therefore it is a good idea to put guardrails in place for yourself, even if your AWS account is just for personal use.
The first place to start is to have an account level billing alarm so you know when your monthly bill is going to be more than what you expected. The next step, if necessary for your work, is to do what we did here and add service level alarms.