Introduction to AWS Budgets, AWS Alarms, AWS Single Sign-On (AWS Single Sign On / AWS SSO), and AWS Cost Anomaly Detection

In this article, we’ll introduce several billing disasters to users of Amazon Web Services (AWS) and then we’ll discuss several tools that can be used to help mitigate this risk including AWS Budgets, AWS Alarms, AWS Single Sign-On (AWS Single Sign On or AWS SSO) and AWS Cost Anomaly Detection.

This really happened and keep in mind that stories such as these seem to happen with some frequency:

“I used my personal AWS account for [an AWS] workshop and thought I had deleted the CF stacks at the end of the workshop, but hadn’t. At the end of the month I was shocked when I saw an unexpected $1500 charge on my AWS bill.” [1]

Here’s another story — this time much worse:

“So, as the old joke goes, an expert is someone that avoids all the small mistakes that amateurs make — to go on and make a really big mistake!

Well, I just discovered a $14k mistake…I had come upon a really nice job request. Automate execution of [AWS] Comprehend-Medical from some json file that ends up in an S3 bucket…Two mistakes were made. (1) As part of my usual flow, I use terraform to test the infrastructure in my own test account. When I ran the test, I underestimated the number of json tags that trigger the automation. (2) Comprehend-Medical is priced rather differently from Comprehend. It’s 20x the cost!” [2]

Sadly these billing disasters may be avoidable had the user simply set up a budget with an alarm that sends an email once the bill reaches a certain threshold.

In this article we’re going to set up a budget with an alarm, and we’re going to briefly look at the Identity and Access Management (IAM) Security Status page, and then we’re going to introduce AWS Single Sign-On (SSO) —  the conclusion along with a short quiz follows.

AWS SSO offers some powerful features that can be used to help mitigate the risk of an account being compromised and we recommend taking some time to read through this section of the article, including the references.

First, let’s look at what happens when your AWS usage skyrockets.

What Happens When Your AWS Usage Spikes?

Stories like those mentioned in the introduction occur with some frequency and when this happens the customer needs to speak with an AWS representative and see if a reprieve can be granted. There are no guarantees that mercy will be shown, however, so it will behoove you to take advantage of a few tools ahead of time to help prevent this kind of surprise from happening in the first place. The following question on Quora should be considered required reading for this article: My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay? — in particular, the response by Chris Clark is instructive:

I use AWS for my popular cocktail site. I was recently hit with a similarly nefarious (albeit not 50K) charge to my account when I added a new service to my stack, and unlike others here, my outcome was not positive. I’m here to represent the less-than-happy path that can occur when this sort of thing happens, offer some advice on how to approach the situation, as well as preventative measures you can take.

The not-so-happy path

Yes, you can file a ticket with billing, and while Amazon is sometimes “good on their customers”, they are not required to be. Ultimately, your claims will be at the mercy of a black-box accounting department and if you don’t like the verdict, you will be told to talk to a lawyer or the police (this is not a joke). Record everything you do and be very deliberate about your moves leading up the the case. They will make it seem like everything is gonna be fine…until it isn’t. AWS does not have the same approach to customer service as other companies. They will eventually just cut you off ice cold, so don’t think the noisy-customer approach that sometimes works with airline companies will work here.

The billing is complex. Don’t file a ticket without all of the data. I dug deep into the root of my charges, and found out I was being charged for 3,000 instance hours a month. How? Well it’s complicated, and it took a long time for me to even find that data, so long that it didn’t matter because my verdict had already been given. Make sure to know everything you can about how and why you were charged so much. They might also ask you to delete any instances related to the billing, which can make it difficult to dig up logs. Consider saving all of the log files before you delete the instances.

Be prepared for anything, because in the end the AWS ToS is a blank check. The AWS billing model is deliberately open-ended, and there is little you can do about it after a verdict has been assigned. You can accumulate a life changing amount of debt on AWS in a matter of days, and unlike your engineering mistakes, this bug can’t be fixed. For this reason, below I will offer advice on how to avoid this situation in the first place.

How you can protect yourself

Set up billing alerts. If you are using AWS, even for a small task, please please please set up billing alerts. They are not required during setup, but if you are a non-enterprise user, I would consider this step mandatory as AWS will not alert you to dramatic increases in charges unless they bypass 15K which is already an incredible amount of money.

Read the pricing table…carefully. If you are installing a new service, make sure to carefully read the pricing table. Amazon will sometimes set ridiculous defaults for container size which you might not see until the bill comes in. Do understand however that this might not be good enough, as bugs, loose API keys, and improper installations can do crazy things.

Consider using another service. If you are a non-business individual user or small-business user, you might want to consider using another service. AWS is built for enterprise customers, and as such an enterprise wallet. Yes, it can be very cheap, but consider this: after my little mix up, I could have payed $150 a month over all of the years I used AWS and still come out ahead. Yes, AWS might be cheap at first, but one mistake can make it very expensive. Currently, AWS does not offer an experience tailored to non-enterprise customers, which can be unbelievably dangerous to new users looking to learn the services. Consider opting for different services with overage models more tailored to protect the individual user.

A feature that appears to be unavailable at the time of this writing, and which is mentioned in the quote above, is the ability to set a hard limit to your budget. This feature request has appeared with some frequency when people have been burned and had they set a budget they may have caught their mistake early and as the saying goes: an ounce of prevention is worth a pound of cure.

Common mistakes that lead to budgetary overruns in AWS

In addition to not having budgets configured, there are two other mistakes people tend to make which can cause problems: they are simply not paying attention to what they’re doing or their personal access tokens have been compromised. If you know of other problems that can result in unexpected budgetary overruns please add them in the comments.

Not Paying Attention

Not paying attention can bite in several ways, for example, the user runs a script that creates many expensive EC2 instances and walks away, a user requires a service and doesn’t check the price first, or a user forgets to delete some data and they learn later that they’re being charged for it and the fees are non-trivial.

Personal Access Keys Compromised

This is a very bad scenario — in this scenario an AWS user includes their personal access keys (PAKs) in a commit that ends up on the Internet. To be clear, we’re not really concerned about the keys so much as the values.

The suggestions here are fairly straightforward:

• Never put credentials in code, full stop. There’s a proper way to set up the AWS environment and you should follow that approach.
• Encrypt the PAKs  — this is important when they’re required for use in a continuous integration platform such as Jenkins. If you need to add the key/value pairs to a server the values should not be stored as cleartext.
• Do not include personal access values in log messages.

GitHub scans for secrets [27] but not everyone uses GitHub and if the scan misses the key/value pairs or if it’s lost by some other means, for example, someone loses a flash drive with the PAKs / PAK values stored as unencrypted text, then this can be a problem.

Create a Budget with an Alarm in the AWS My Billing Dashboard 

In this section, we’ll create a budget of $250, and when 80% of the budget is reached an alarm will sound and an email will be sent.

The first image shows us that we can find the Budgets page under the “My Billing Dashboard” menu item.

In the following image, we can see where the “Create budget” button is located. Note that in this example we already have an existing budget which is set at $100.00 (USD).

The next page allows us to choose what kind of budget type this will be — in this example we’ll choose the “Cost budget”.

The following page facilitates the creation of the budget and we’ve gone into detail in the image regarding what the important aspects of this page include, along with an explanation pertaining to the options available in the “Aggregate costs by” option.

See also: Understanding your AWS Cost Datasets: A Cheat Sheet

 

The next step in this process involves creating an alert, which is shown below.

Finally, we review the budget and then create it.

 

We can see that our new budget now appears on the “AWS Budgets” page.

 

Requires further research: If a budget is exceeded, when exactly is the notification sent? We should not assume that an alarm causes the notification to be sent immediately unless there’s evidence to support this. If you find this information please add it in a comment with the link to the source.

In the next section, we’ll briefly explore the IAM Security Status, we’ll then introduce AWS SSO, and finally, we’ll conclude the article and we hope that you’ll comment on the quiz that follows.