Introduction to AWS Budgets, AWS Alarms, AWS Single Sign-On (AWS Single Sign On / AWS SSO), and AWS Cost Anomaly Detection: Protect yourself from unexpectedly high AWS bills

In this article, we’ll introduce several billing disasters to users of Amazon Web Services (AWS) and then we’ll discuss several tools that can be used to help mitigate this risk including AWS Budgets, AWS Alarms, AWS Single Sign-On (AWS Single Sign On or AWS SSO) and AWS Cost Anomaly Detection.

This really happened and keep in mind that stories such as these seem to happen with some frequency:

“I used my personal AWS account for [an AWS] workshop and thought I had deleted the CF stacks at the end of the workshop, but hadn’t. At the end of the month I was shocked when I saw an unexpected $1500 charge on my AWS bill.” [1]

Here’s another story — this time much worse:

“So, as the old joke goes, an expert is someone that avoids all the small mistakes that amateurs make — to go on and make a really big mistake!

Well, I just discovered a $14k mistake…I had come upon a really nice job request. Automate execution of [AWS] Comprehend-Medical from some json file that ends up in an S3 bucket…Two mistakes were made. (1) As part of my usual flow, I use terraform to test the infrastructure in my own test account. When I ran the test, I underestimated the number of json tags that trigger the automation. (2) Comprehend-Medical is priced rather differently from Comprehend. It’s 20x the cost!” [2]

Sadly these billing disasters may be avoidable had the user simply set up a budget with an alarm that sends an email once the bill reaches a certain threshold.

In this article we’re going to set up a budget with an alarm, and we’re going to briefly look at the Identity and Access Management (IAM) Security Status page, and then we’re going to introduce AWS Single Sign-On (SSO) — the conclusion along with a short quiz follows.

AWS SSO offers some powerful features that can be used to help mitigate the risk of an account being compromised and we recommend taking some time to read through this section of the article, including the references.

First, let’s look at what happens when your AWS usage skyrockets.

What Happens When Your AWS Usage Spikes?

Stories like those mentioned in the introduction occur with some frequency and when this happens the customer needs to speak with an AWS representative and see if a reprieve can be granted. There are no guarantees that mercy will be shown, however, so it will behoove you to take advantage of a few tools ahead of time to help prevent this kind of surprise from happening in the first place. The following question on Quora should be considered required reading for this article: My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay? — in particular, the response by Chris Clark is instructive:

I use AWS for my popular cocktail site. I was recently hit with a similarly nefarious (albeit not 50K) charge to my account when I added a new service to my stack, and unlike others here, my outcome was not positive. I’m here to represent the less-than-happy path that can occur when this sort of thing happens, offer some advice on how to approach the situation, as well as preventative measures you can take.

The not-so-happy path

Yes, you can file a ticket with billing, and while Amazon is sometimes “good on their customers”, they are not required to be. Ultimately, your claims will be at the mercy of a black-box accounting department and if you don’t like the verdict, you will be told to talk to a lawyer or the police (this is not a joke). Record everything you do and be very deliberate about your moves leading up the the case. They will make it seem like everything is gonna be fine…until it isn’t. AWS does not have the same approach to customer service as other companies. They will eventually just cut you off ice cold, so don’t think the noisy-customer approach that sometimes works with airline companies will work here.

The billing is complex. Don’t file a ticket without all of the data. I dug deep into the root of my charges, and found out I was being charged for 3,000 instance hours a month. How? Well it’s complicated, and it took a long time for me to even find that data, so long that it didn’t matter because my verdict had already been given. Make sure to know everything you can about how and why you were charged so much. They might also ask you to delete any instances related to the billing, which can make it difficult to dig up logs. Consider saving all of the log files before you delete the instances.

Be prepared for anything, because in the end the AWS ToS is a blank check. The AWS billing model is deliberately open-ended, and there is little you can do about it after a verdict has been assigned. You can accumulate a life changing amount of debt on AWS in a matter of days, and unlike your engineering mistakes, this bug can’t be fixed. For this reason, below I will offer advice on how to avoid this situation in the first place.

How you can protect yourself

Set up billing alerts. If you are using AWS, even for a small task, please please please set up billing alerts. They are not required during setup, but if you are a non-enterprise user, I would consider this step mandatory as AWS will not alert you to dramatic increases in charges unless they bypass 15K which is already an incredible amount of money.

Read the pricing table…carefully. If you are installing a new service, make sure to carefully read the pricing table. Amazon will sometimes set ridiculous defaults for container size which you might not see until the bill comes in. Do understand however that this might not be good enough, as bugs, loose API keys, and improper installations can do crazy things.

Consider using another service. If you are a non-business individual user or small-business user, you might want to consider using another service. AWS is built for enterprise customers, and as such an enterprise wallet. Yes, it can be very cheap, but consider this: after my little mix up, I could have payed $150 a month over all of the years I used AWS and still come out ahead. Yes, AWS might be cheap at first, but one mistake can make it very expensive. Currently, AWS does not offer an experience tailored to non-enterprise customers, which can be unbelievably dangerous to new users looking to learn the services. Consider opting for different services with overage models more tailored to protect the individual user.

A feature that appears to be unavailable at the time of this writing, and which is mentioned in the quote above, is the ability to set a hard limit to your budget. This feature request has appeared with some frequency when people have been burned and had they set a budget they may have caught their mistake early and as the saying goes: an ounce of prevention is worth a pound of cure.

Common mistakes that lead to budgetary overruns in AWS

In addition to not having budgets configured, there are two other mistakes people tend to make which can cause problems: they are simply not paying attention to what they’re doing or their personal access tokens have been compromised. If you know of other problems that can result in unexpected budgetary overruns please add them in the comments.

Not Paying Attention

Not paying attention can bite in several ways, for example, the user runs a script that creates many expensive EC2 instances and walks away, a user requires a service and doesn’t check the price first, or a user forgets to delete some data and they learn later that they’re being charged for it and the fees are non-trivial.

Personal Access Keys Compromised

This is a very bad scenario — in this scenario an AWS user includes their personal access keys (PAKs) in a commit that ends up on the Internet. To be clear, we’re not really concerned about the keys so much as the values.

The suggestions here are fairly straightforward:

Never put credentials in code, full stop. There’s a proper way to set up the AWS environment and you should follow that approach.
Encrypt the PAKs — this is important when they’re required for use in a continuous integration platform such as Jenkins. If you need to add the key/value pairs to a server the values should not be stored as cleartext.
Do not include personal access values in log messages.

GitHub scans for secrets [27] but not everyone uses GitHub and if the scan misses the key/value pairs or if it’s lost by some other means, for example, someone loses a flash drive with the PAKs / PAK values stored as unencrypted text, then this can be a problem.

How To Create An AWS Budget With An Alarm In The AWS My Billing Dashboard

In this section, we’ll create a budget of $250, and when 80% of the budget is reached an alarm will sound and an email will be sent.

Step One: Find the Budgets page under the “My Billing Dashboard”

The first image shows us that we can find the Budgets page under the “My Billing Dashboard” menu item.

Step Two: Press the Create AWS Budget button

In the following image, we can see where the “Create budget” button is located. Note that in this example we already have an existing budget which is set at $100.00 (USD).

Step Three: Set the budget type to “Cost Budget”

The next page allows us to choose what kind of budget type this will be — in this example, we’ll choose the “Cost budget”.

Step Four: Set Your AWS Budget

The following page facilitates the creation of the budget and we’ve gone into detail in the image regarding what the important aspects of this page include, along with an explanation pertaining to the options available in the “Aggregate costs by” option.

Step Five: Create An Alert

The next step in this process involves creating an alert, which is shown below.

Step Six: Review The AWS Budget and Create It

Finally, we review the AWS budget and if everything looks correct then we go ahead and create it.

Step Seven: Confirm That The Budget Has Been Created

We can see that our new budget now appears on the “AWS Budgets” page.

Requires further research: If a budget is exceeded, when exactly is the notification sent? We should not assume that an alarm causes the notification to be sent immediately unless there’s evidence to support this. If you find this information please add it in a comment with the link to the source.

In the next section, we’ll briefly explore the IAM Security Status, we’ll then introduce AWS SSO, and finally, we’ll conclude the article and we hope that you’ll comment on the quiz that follows.

AWS IAM Security Status

If you are new to AWS you should take a look at the Identity and Access Management (IAM)console and if it doesn’t look like what we have below then consider addressing whatever flags that have been raised.

Further instructions regarding how to achieve the aforementioned results can be found here:

In the next section, we’ll introduce AWS Single Sign-On (SSO).

AWS Single Sign-On (AWS Single Sign On / SSO)

AWS SSO is free [13, 22, 23] and requires AWS Organizations, which is also available at no additional charge, and which provides the AWS account owner with better control over access to AWS resources. We’ve included several resources for learning about AWS SSO and we’re not going to repeat that here however please see the references section of this article below, specifically numbers #13 through #24.

In the test setup that we configured, we set the session time for the Developers group to one hour and we also restricted the policy to AmazonEC2FullAccess. We authenticated via the URL provided to us by Amazon at which point we could select the environment variables from the command line or programmatic access link, see the image below.

We set these variables in the ~/.aws/credentials file and then executed the following via the command line:

aws ec2 run-instances --image-id ami-02354e95b39ca8dec --count 1 --instance-type t2.micro --region us-east-1 --profile 221025111773_AWS_1HR_AmazonEC2FullAccess_PS --subnet subnet-cm235nh7cc7296312 --tag-specifications 'ResourceType=instance,Tags=[{Key=costCenter,Value=45678}, {Key=department,Value=Energy}]'

And the EC2 instance was created.

If we attempt to run the same command after an hour has elapsed, we’ll see the following message:

An error occurred (RequestExpired) when calling the RunInstances operation: Request has expired.

It is helpful to note a few things here:

The user authenticates with a username, a password, and a multi-factor authentication (MFA) code.
The session only lasts for one hour.
The credentials automatically expire.
The user is limited to the Identity and Access Management (IAM) Policy permissions, in this case, AmazonEC2FullAccess.
Activity is tracked via CloudWatch.
This is slightly more complicated than using personal access keys however it only took about fifteen minutes to set up and test.

The last point we’ll make about this involves AWS Systems Manager Session Manager (SMSM) and is left as an exercise for the reader — quoting from [25]:

“Session Manager provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys…while still providing end users with simple one-click cross-platform access to your managed instances.”

The exercise we alluded to is as follows: Demonstrate the usage of SMSM to run a single command, such as “ls”, on a running EC2 instance.‎

AWS Budgets, AWS Alarms, AWS Single Sign-On, and AWS Cost Anomaly Detection Conclusion

While researching this article we found a new AWS service which is in preview at the moment called AWS Cost Anomaly Detection (CAD) — from [29]:

Starting today, customers can use AWS Cost Anomaly Detection to detect unexpected or unusual spend. AWS Cost Anomaly Detection uses a multi-layered state machine learning model that learns your unique spend patterns to adjust spend thresholds — this means you do not need to worry about determining appropriate thresholds (e.g. 10% increase in daily spend) and maintaining them as your usage changes over time. These machine learning layers allow Anomaly Detection to detect various types of anomalies, such as a one-time cost spike or gradual, consistent cost increases. Anomaly Detection is customizable to segment your spend data and provides tailored alerting preferences so that you are informed as soon as an anomaly is detected.

This service preview was released on the 24th of September 2020 and we’ve yet to experiment with it however this sounds like it should be on everyone’s radar.

We’ll close this article by suggesting a few things:

Set a budget and an alarm.
Pay attention to what you’re doing.
If you’re not using AWS SSO then consider doing so.
If you create users in AWS and at some point in the future the user isn’t being used, disable the user by removing all policies; or consider deleting that user altogether.
The same logic applies to personal access keys — if they’re not being used, deactivate them.
Secure your account(s) according to recommendations provided by the IAM Security Status page.
The quiz below contains further information that can be helpful toward securing your AWS account.

I’ve written a supporting article that answers the question “What Is The Purpose Of Cost Allocation Tags in AWS?” which you might helpful as well — this post includes a fake case study about Company X and how the business was able to save money using AWS Cost Allocation Tags. I also have an another article entitled “AWS Cost Allocation Tags: Lower AWS Billing Cost Now!” which covers how to use AWS Cost Allocation Tags to help lower AWS billing charges.

Hopefully, this article has been helpful. If you have any questions, please leave them in the comments section and we can discuss them. Take the quiz!

Happy coding!

Hat tip: Allan Webb, Timothy Ingalls