Introduction to AWS Budgets, AWS Alarms, AWS Single Sign-On (AWS Single Sign On / AWS SSO), and AWS Cost Anomaly Detection: Protect yourself from unexpectedly high AWS bills
In this article, we’ll introduce several billing disasters to users of Amazon Web Services (AWS) and then we’ll discuss several tools that can be used to help mitigate this risk including AWS Budgets, AWS Alarms, AWS Single Sign-On (AWS Single Sign On or AWS SSO) and AWS Cost Anomaly Detection.
This really happened and keep in mind that stories such as these seem to happen with some frequency:
âI used my personal AWS account for [an AWS] workshop and thought I had deleted the CF stacks at the end of the workshop, but hadnât. At the end of the month I was shocked when I saw an unexpected $1500 charge on my AWS bill.â [1]
Hereâs another storyâââthis time much worse:
âSo, as the old joke goes, an expert is someone that avoids all the small mistakes that amateurs makeâââto go on and make a really big mistake!
Well, I just discovered a $14k mistakeâĻI had come upon a really nice job request. Automate execution of [AWS] Comprehend-Medical from some json file that ends up in an S3 bucketâĻTwo mistakes were made. (1) As part of my usual flow, I use terraform to test the infrastructure in my own test account. When I ran the test, I underestimated the number of json tags that trigger the automation. (2) Comprehend-Medical is priced rather differently from Comprehend. Itâs 20x the cost!â [2]
Sadly these billing disasters may be avoidable had the user simply set up a budget with an alarm that sends an email once the bill reaches a certain threshold.
In this article weâre going to set up a budget with an alarm, and weâre going to briefly look at the Identity and Access Management (IAM) Security Status page, and then weâre going to introduce AWS Single Sign-On (SSO)âââ the conclusion along with a short quiz follows.
AWS SSO offers some powerful features that can be used to help mitigate the risk of an account being compromised and we recommend taking some time to read through this section of the article, including the references.
First, letâs look at what happens when your AWS usage skyrockets.
What Happens When Your AWS Usage Spikes?
Stories like those mentioned in the introduction occur with some frequency and when this happens the customer needs to speak with an AWS representative and see if a reprieve can be granted. There are no guarantees that mercy will be shown, however, so it will behoove you to take advantage of a few tools ahead of time to help prevent this kind of surprise from happening in the first place. The following question on Quora should be considered required reading for this article: My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay?âââin particular, the response by Chris Clark is instructive:
I use AWS for my popular cocktail site. I was recently hit with a similarly nefarious (albeit not 50K) charge to my account when I added a new service to my stack, and unlike others here, my outcome was not positive. Iâm here to represent the less-than-happy path that can occur when this sort of thing happens, offer some advice on how to approach the situation, as well as preventative measures you can take.
The not-so-happy path
Yes, you can file a ticket with billing, and while Amazon is sometimes âgood on their customersâ, they are not required to be. Ultimately, your claims will be at the mercy of a black-box accounting department and if you donât like the verdict, you will be told to talk to a lawyer or the police (this is not a joke). Record everything you do and be very deliberate about your moves leading up the the case. They will make it seem like everything is gonna be fineâĻuntil it isnât. AWS does not have the same approach to customer service as other companies. They will eventually just cut you off ice cold, so donât think the noisy-customer approach that sometimes works with airline companies will work here.
The billing is complex. Donât file a ticket without all of the data. I dug deep into the root of my charges, and found out I was being charged for 3,000 instance hours a month. How? Well itâs complicated, and it took a long time for me to even find that data, so long that it didnât matter because my verdict had already been given. Make sure to know everything you can about how and why you were charged so much. They might also ask you to delete any instances related to the billing, which can make it difficult to dig up logs. Consider saving all of the log files before you delete the instances.
Be prepared for anything, because in the end the AWS ToS is a blank check. The AWS billing model is deliberately open-ended, and there is little you can do about it after a verdict has been assigned. You can accumulate a life changing amount of debt on AWS in a matter of days, and unlike your engineering mistakes, this bug canât be fixed. For this reason, below I will offer advice on how to avoid this situation in the first place.
How you can protect yourself
Set up billing alerts. If you are using AWS, even for a small task, please please please set up billing alerts. They are not required during setup, but if you are a non-enterprise user, I would consider this step mandatory as AWS will not alert you to dramatic increases in charges unless they bypass 15K which is already an incredible amount of money.
Read the pricing tableâĻcarefully. If you are installing a new service, make sure to carefully read the pricing table. Amazon will sometimes set ridiculous defaults for container size which you might not see until the bill comes in. Do understand however that this might not be good enough, as bugs, loose API keys, and improper installations can do crazy things.
Consider using another service. If you are a non-business individual user or small-business user, you might want to consider using another service. AWS is built for enterprise customers, and as such an enterprise wallet. Yes, it can be very cheap, but consider this: after my little mix up, I could have payed $150 a month over all of the years I used AWS and still come out ahead. Yes, AWS might be cheap at first, but one mistake can make it very expensive. Currently, AWS does not offer an experience tailored to non-enterprise customers, which can be unbelievably dangerous to new users looking to learn the services. Consider opting for different services with overage models more tailored to protect the individual user.
A feature that appears to be unavailable at the time of this writing, and which is mentioned in the quote above, is the ability to set a hard limit to your budget. This feature request has appeared with some frequency when people have been burned and had they set a budget they may have caught their mistake early and as the saying goes: an ounce of prevention is worth a pound of cure.
Common mistakes that lead to budgetary overruns in AWS
In addition to not having budgets configured, there are two other mistakes people tend to make which can cause problems: they are simply not paying attention to what theyâre doing or their personal access tokens have been compromised. If you know of other problems that can result in unexpected budgetary overruns please add them in the comments.
Not Paying Attention
Not paying attention can bite in several ways, for example, the user runs a script that creates many expensive EC2 instances and walks away, a user requires a service and doesnât check the price first, or a user forgets to delete some data and they learn later that theyâre being charged for it and the fees are non-trivial.
Personal Access Keys Compromised
This is a very bad scenarioâââin this scenario an AWS user includes their personal access keys (PAKs) in a commit that ends up on the Internet. To be clear, weâre not really concerned about the keys so much as the values.
The suggestions here are fairly straightforward:
- Never put credentials in code, full stop. Thereâs a proper way to set up the AWS environment and you should follow that approach.
- Encrypt the PAKs âââthis is important when theyâre required for use in a continuous integration platform such as Jenkins. If you need to add the key/value pairs to a server the values should not be stored as cleartext.
- Do not include personal access values in log messages.
GitHub scans for secrets [27] but not everyone uses GitHub and if the scan misses the key/value pairs or if itâs lost by some other means, for example, someone loses a flash drive with the PAKs / PAK values stored as unencrypted text, then this can be a problem.
How To Create An AWS Budget With An Alarm In The AWS My Billing Dashboard
In this section, weâll create a budget of $250, and when 80% of the budget is reached an alarm will sound and an email will be sent.
Step One: Find the Budgets page under the âMy Billing Dashboardâ
The first image shows us that we can find the Budgets page under the âMy Billing Dashboardâ menu item.

Step Two: Press the Create AWS Budget button
In the following image, we can see where the âCreate budgetâ button is located. Note that in this example we already have an existing budget which is set at $100.00 (USD).

Step Three: Set the budget type to “Cost Budget”
The next page allows us to choose what kind of budget type this will beâââin this example, weâll choose the âCost budgetâ.

Step Four: Set Your AWS Budget
The following page facilitates the creation of the budget and weâve gone into detail in the image regarding what the important aspects of this page include, along with an explanation pertaining to the options available in the âAggregate costs byâ option.
See also: Understanding your AWS Cost Datasets: A Cheat Sheet

Step Five: Create An Alert
The next step in this process involves creating an alert, which is shown below.

Step Six: Review The AWS Budget and Create It
Finally, we review the AWS budget and if everything looks correct then we go ahead and create it.

Step Seven: Confirm That The Budget Has Been Created
We can see that our new budget now appears on the âAWS Budgetsâ page.

Requires further research: If a budget is exceeded, when exactly is the notification sent? We should not assume that an alarm causes the notification to be sent immediately unless thereâs evidence to support this. If you find this information please add it in a comment with the link to the source.
In the next section, weâll briefly explore the IAM Security Status, weâll then introduce AWS SSO, and finally, weâll conclude the article and we hope that youâll comment on the quiz that follows.
AWS IAM Security Status
If you are new to AWS you should take a look at the Identity and Access Management (IAM)console and if it doesnât look like what we have below then consider addressing whatever flags that have been raised.

Further instructions regarding how to achieve the aforementioned results can be found here:
- AWS account root user
- Enabling a virtual multi-factor authentication (MFA) device (console)
- Creating an IAM user in your AWS account
- Security best practices in IAM
- Setting an account password policy for IAM users
In the next section, weâll introduce AWS Single Sign-On (SSO).
AWS Single Sign-On (AWS Single Sign On / SSO)
AWS SSO is free [13, 22, 23] and requires AWS Organizations, which is also available at no additional charge, and which provides the AWS account owner with better control over access to AWS resources. Weâve included several resources for learning about AWS SSO and weâre not going to repeat that here however please see the references section of this article below, specifically numbers #13 through #24.
In the test setup that we configured, we set the session time for the Developers group to one hour and we also restricted the policy to AmazonEC2FullAccess. We authenticated via the URL provided to us by Amazon at which point we could select the environment variables from the command line or programmatic access link, see the image below.

We set these variables in the ~/.aws/credentials file and then executed the following via the command line:
aws ec2 run-instances --image-id ami-02354e95b39ca8dec --count 1 --instance-type t2.micro --region us-east-1 --profile 221025111773_AWS_1HR_AmazonEC2FullAccess_PS --subnet subnet-cm235nh7cc7296312 --tag-specifications 'ResourceType=instance,Tags=[{Key=costCenter,Value=45678}, {Key=department,Value=Energy}]'
And the EC2 instance was created.
If we attempt to run the same command after an hour has elapsed, weâll see the following message:
An error occurred (RequestExpired) when calling the RunInstances operation: Request has expired.
It is helpful to note a few things here:
- The user authenticates with a username, a password, and a multi-factor authentication (MFA) code.
- The session only lasts for one hour.
- The credentials automatically expire.
- The user is limited to the Identity and Access Management (IAM) Policy permissions, in this case, AmazonEC2FullAccess.
- Activity is tracked via CloudWatch.
- This is slightly more complicated than using personal access keys however it only took about fifteen minutes to set up and test.
The last point weâll make about this involves AWS Systems Manager Session Manager (SMSM) and is left as an exercise for the readerâââquoting from [25]:
âSession Manager provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keysâĻwhile still providing end users with simple one-click cross-platform access to your managed instances.â
The exercise we alluded to is as follows: Demonstrate the usage of SMSM to run a single command, such as âlsâ, on a running EC2 instance.â
AWS Budgets, AWS Alarms, AWS Single Sign-On, and AWS Cost Anomaly Detection Conclusion
While researching this article we found a new AWS service which is in preview at the moment called AWS Cost Anomaly Detection (CAD)âââfrom [29]:
Starting today, customers can use AWS Cost Anomaly Detection to detect unexpected or unusual spend. AWS Cost Anomaly Detection uses a multi-layered state machine learning model that learns your unique spend patterns to adjust spend thresholdsâââthis means you do not need to worry about determining appropriate thresholds (e.g. 10% increase in daily spend) and maintaining them as your usage changes over time. These machine learning layers allow Anomaly Detection to detect various types of anomalies, such as a one-time cost spike or gradual, consistent cost increases. Anomaly Detection is customizable to segment your spend data and provides tailored alerting preferences so that you are informed as soon as an anomaly is detected.
This service preview was released on the 24th of September 2020 and weâve yet to experiment with it however this sounds like it should be on everyoneâs radar.
Weâll close this article by suggesting a few things:
- Set a budget and an alarm.
- Pay attention to what youâre doing.
- If youâre not using AWS SSO then consider doing so.
- If you create users in AWS and at some point in the future the user isnât being used, disable the user by removing all policies; or consider deleting that user altogether.
- The same logic applies to personal access keysâââif theyâre not being used, deactivate them.
- Secure your account(s) according to recommendations provided by the IAM Security Status page.
- The quiz below contains further information that can be helpful toward securing your AWS account.
Iâve written a supporting article that answers the question âWhat Is The Purpose Of Cost Allocation Tags in AWS?â which you might helpful as well â this post includes a fake case study about Company X and how the business was able to save money using AWS Cost Allocation Tags. I also have an another article entitled “AWS Cost Allocation Tags: Lower AWS Billing Cost Now!” which covers how to use AWS Cost Allocation Tags to help lower AWS billing charges.
Hopefully, this article has been helpful. If you have any questions, please leave them in the comments section and we can discuss them. Take the quiz!
Happy coding!
Hat tip: Allan Webb, Timothy Ingalls
References
- Anonymized conversation
- Anonymized conversation
- I was billed for 14k USD on Amazon Web Services (note this source is not the same as #1 even though the numbers are equal)
- Getting Started: Follow Security Best Practices as You Configure Your AWS Resources
- Top 10 security items to improve in your AWS account
- Ryan Hellyerâs AWS Nightmare: Leaked Access Keys Result in a $6,000 Bill Overnight
- My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay?
- AWS Cost Allocation Tags and Cost Reduction:
Optimize your AWS services to pay less per month with cost allocation tags - Understanding your AWS Cost Datasets: A Cheat Sheet
- Enabling a virtual multi-factor authentication (MFA) device (console)
- How can I use IAM roles to restrict API calls from specific IP addresses to the AWS Management Console?
- Storing Secrets
- AWS Single Sign-On (SSO)
- Introducing AWS Single Sign-On
- How to retrieve short-term credentials for CLI use with AWS Single Sign-on
- Configuring the AWS CLI to use AWS Single Sign-On
- Identity and Access Management for AWS SSO
- Overview of Managing Access Permissions to Your AWS SSO Resources
- Requesting temporary security credentials
- Amazon WorkSpaces Cost Optimizer
- AWS Single Sign-On Features
- AWS Single Sign-On FAQs
- AWS Organizations FAQs
- How to create and manage users within AWS Single Sign-On
- AWS Systems Manager Session Manager (SMSM)
- (SMSM) Start a session
- GitHub About secret scanning
- My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay?
- Preview: Anomaly Detection and alerting now available in AWS Cost Management
- AWS Cost Allocation Tags and Cost Reduction
You must log in to post a comment.