Aviram Levy, head of the product enablement at Zesty, shares a detailed overview of the storage services available on the AWS Cloud platform. After the introduction, the pricing of each storage service will be provided to the reader as specified by Amazon. Aviram further lists the hidden culprits contributing to unexplained increments in the cost of running storage services on AWS. Where possible, these culprits will be exposed in detail to the reader, along with their added cost.
Cloud computing is an innovative technology, but the cost of working in the cloud can be more than we expect sometimes. Despite the growing number of cloud service providers, engineering teams experience unstable prices for the same monthly services, leading to 35% of wasted cloud spendOpens a new window .
Enterprise organizations, in particular, are feeling this pain. Pinterest experienced a 41% higher AWS bill in 2018 versus 2017Opens a new window due to having to purchase extra capacity at a higher cost despite the fact they had paid for capacity in advance. Furthermore, a 2021 report by VirtanaOpens a new window indicated that 82% of 350 companies based in the US and UK experience â€œunnecessaryâ€ cloud costs.â€ The same report noted a Gartner prediction that â€œ60 percent of infrastructure and operations leaders will encounter cloud cost overruns by 2024.â€
There are multiple cases, reported or not, where unnecessary cloud spend burns holes in a company’s technical purse.Â
This article will help you identify the culprits responsible for cloud waste on AWS and teach you how to nip them in the bud. We’ll also discuss how S3 pricing is calculated and some general cost optimization tenets.
Brief Overview of AWS Storage Services
On the AWS cloud platform, customers can take advantage of a pay-as-you-go model, meaning customers only pay for the individual resources and services they consume for a period of time.
The price of using AWS’ state-of-the-art cloud tools depends on three factors: computing, storage, and outbound data transfers. In this article, we’ll discuss storage services with hidden costs, which we’ll classify as our primaryÂ â€œculprits.â€ Several AWS storage services include
- Amazon Simple Storage Services (S3)
- Amazon Elastic Block Store (EBS)
- Amazon Elastic File System (EFS)
- Amazon FSx for Lustre
- Amazon S3 Glacier
- Amazon FSx for Windows File Server
- AWS Storage Gateway
However, as S3 and EBS are the most commonly used, we will focus on them when discussing AWS culprits below. Also, this article will dive deeper into Amazon S3 since it has become the standard storage service for developers due to being more cost-efficient and durable.
Amazon S3 Pricing
It’s essential to know that for the S3 service, you only pay for the storage capacity you use, not the capacity provisioned. In other services, you pay for the resources provisioned, whether or not you used them.
Typically, there are six cost components for the S3 service:
- Storage: You pay for storing objects in S3 buckets.
- Requests and data retrievals: Your cost depends on the type of requests made against your S3 buckets, with DELETE and CANCEL exceptions.
- Data transfer: You pay for the bandwidths that come in and go out of the S3 service, with some exceptions.
- Management and analytics: You pay for storage management features and enabled analytics.
- Replication: Your cost depends on the class of region replication you choose.
- S3 Object Lambda for processing data: An S3 Object Lambda lets your S3 GET request call a defined AWS Lambda function to process data and send it back as a processed object to your app.
How S3 Cost Is Calculated
AWS S3 service costs can be grouped into three major categories:
- The standard storage service is charged hourly in Gigabytes per month, at $0.023/GB/month for the first 50 Terabytes; this decreases as you consume more Terabytes of storage.
- The standard API costs for file operations per 1,000 requests is $0.0004 for read requests and $0.005 (10x more) for write requests.
- The standard cost of transferring data outside an AWS region is $0.02/GB/month and $0.09 per Gigabyte to the internet for the first 10 Terabytes; this decreases as you transfer more Terabytes of storage.
S3 Culprits and How to Optimize Them
There are several hidden culprits and techniques for optimizing AWS S3 cloud spend.
Compute and storage location
Having an Amazon EC2 instance and your S3 service set to different AWS regions reduces the performance overhead and increases the data transfer cost. You can transfer data for free when the EC2 instance and the S3 bucket are set to the same region.
If your data transfer usually includes another region, replicating the S3 bucket might be the best way to optimize spend. Remember that replicating the bucket may become expensive if the objects and total size are large or there are too many replications. Also, replication may lead to inconsistency in data. If there is high traffic in downloads from the S3 servers, migrate cacheable files to AWS CloudFront since it is more cost-effective and performant than S3 in most cases.
Setting up CloudWatch alerts is important for monitoring usage metrics. Furthermore, you can also enable Cost Explorer or download reports about your S3 buckets to stay informed about your usage costs.
Keeping irrelevant files
Remember to delete files when they are no longer relevant. Also, if you have any relevant files that are not currently needed but can be recreated with relative ease, delete them too. Confirm that there are no incomplete uploads; if there are, delete them since if you leave them, you pay for them. TheÂ S3 lifecycleOpens a new window helps you automate this process by enabling you to configure the lifecycle per various policies. In addition, this feature also lets you retain data at a reduced cost.Â
AWS charges for every version of a data object that is stored and transferred. Replications of an object are complete objects by themselves. If you have multiple versions of an object, you will be billed for them all.Â
Compressing objects deliver better performance, reduce the storage capacity required, and, in turn, lowers costs. Depending on the use case, compressions such as LZ4, GZIP, or even ZSTD provide good performance and take up minimal space.
API calls remain the same.
Whether you’re uploading a large or a smaller object, the API cost is not dependent on the data size but on the type of call. That means you can only optimize it by uploading in bulk instead of in chunks, so you may have to batch or zip different files to create a single file for upload.
S3 intelligent tiering
Transitioning objects from the standard S3 service to the S3 Intelligent Tiering service attracts an additional fee. Depending on the access patterns of your objects, you may or may not be saving costs by using Intelligent Tiering.Â
S3 Storage Tier
|Cost per Gigabytes Monthly||
Discount Relative to Standard Storage
|Deep Archive Access||$0.00099||
Data stored in Infrequent Access (IA) requires a retrieval fee of $0.01 per GB, but this is 4x cheaper than using S3 storage even though retrieval is free there. IA is the best option if you retrieve your data objects once a month. Also, you can use IA for disaster recovery.Â
Data stored in the Frequent Access tier doesn’t save any cloud costs. It is noteworthy that S3 has a monitoring fee of $0.0025 per 1,000 data objects in a month. The formula for this is:
Monitoring costs per month = (number of objects with Intelligent Tiering/1000 objects) * $0.0025.
Note that transitioning objects to the lower tiers come with tradeoffs like reduced replications and SLA and a longer retrieval time for the lowest tiers. You should also note that there is a fee for transitioning data objects between two storage classes at a rate of $0.01 per 1,000 objects:
Transition costs per month = (number of data objects being transitioned / 1000 objects) * $0.01
AWS key management system alternative
Your cloud bill increases when you use Amazon S3 with the AWS KMS service because every data object requires a KMS encryption operation to write and a decryption operation to read. To reduce this cost, use the Amazon S3 Bucket Keys instead. This provides as much as a 99% reduction in unnecessary spending without changing dependent applications.
It is worth knowing that you are billed for disabled KMS keys on your account and active keys. So, consider deleting disabled keys if they don’t adversely affect services configured with those keys.
Amazon Elastic Block Store (EBS)
Amazon EBS comprises volume storage and snapshots. The storage volumes can be attached to EC2 instances to carry out block storage tasks. The snapshots provide an easier and more secure way to protect stored data such as data volumes, boot volumes, non-EBS volumes, and block data. For this reason, the pricing for EBS volumes is different from that of EBS snapshots.
The standard cost for snapshots is $0.05/GB/month. The cost for the archive is $0.0125/GB/month.Â
Optimizing AWS EBS culprits
Optimizing your AWS EBS cost is a straightforward process that can be categorized into two parts.
Deleting independent volumes and old snapshots
Deleting a volume significantly reduces your cloud storage spend since EBS volumes are priced based on the amount of storage used. Contrarily, deleting a snapshot might not reduce cloud storage spending since only the data referenced by the snapshot is deleted.
Even after stopping an instance, the volumes configured to those instances retain information and add it to your bill. To stop EBS charges, consider deleting any volume or snapshot you’re not using.
Emptying the recycle bin
You pay for the EBS storage capacity provisioned to you. This can be either a volume, a snapshot, or in the recycle bin. The recycle bin secures snapshots that were deleted unintentionally. For this reason, snapshots in the recycle bin cost the same as a normal snapshot. So remember to delete the contents of the recycle bin to confirm deleted snapshots.
Giving inexperienced users access to perform operations with the S3 service can also warrant unnecessary costs. Many developers have spawned demo instances but failed to turn them off after usage. It is advisable to set down and follow procedures while working with cloud infrastructure and promote a culture of ownership towards created resources.
When an AWS service is not being used for a long time, it’s always best to turn it off or delete it if possible, as it may be contributing to wastage without your knowledge.
Define and use monitoring metrics where possible. Also, endeavor to check these metrics regularly to avoid surprises. You can leverage alerts and notifications frequently to avoid waking up to outrageous bills. Building and training a team to architect and manage your cloud spend might be a good idea. Finally, a culture of retention-for-everything and backup in lower S3 tiers is beneficial to implement from the beginning, as it’s sometimes impossible to tell which data object is useful in the long term.
How do you think organizations can reduce the aforementioned costs while using the storage services? Share your thoughts with us on FacebookOpens a new window , TwitterOpens a new window , and LinkedInOpens a new window . We’d love to hear from you!