It’s a typical business scenario. A three-day weekend and a four-day break beckons, thanks to an imminent bank holiday, and in their eagerness to start the long weekend one of your software developers forgets to turn off the 25 server cluster on-demand instance they were renting on Friday morning. So while Sam the Space Cadet spent the weekend happily consuming chocolate, that cluster spent the weekend happily consuming money. It may not have been doing anything, but anybody who has taken a cab knows what happens when it’s idling with the meter running.
“The key to cloud economics is to not buy cloud services in the same way as you do traditional IT,” observes James Staten, an infrastructure and operations analyst at Forrester Research. “Consume only what you need when you need it and be diligent about turning it off.”
Cloud vendors now offer various metrics and alarms that can generate email, SMS and other alerts when a service or account approaches or exceeds a cost threshold or usage tier.
These types of alerts go some way to helping measure and manage usage, but if you are going to use cloud infrastructure resources from the likes of Amazon, Google and Microsoft, you have to consider the cost of many things you may not even think about with traditional IT. These range from the amount of CPU time used through instance types to SQL reporting hours. The matrix of options and pricing schedules is vast and costs can seem opaque, whether you are trying to estimate future use or understand the information in your bill.
Simplifying costs – slightly
“We have had a number of requests for better access to more detailed billing data,” acknowledges Jeff Barr, chief evangelist for Amazon Web Services (AWS). So have other cloud infrastructure providers, because the granularity of cloud infrastructure services has developed much faster than the granularity of the data that service providers make available on the costs for consumption and provisioning.
Let’s be realistic though. It’s not in the best interests of providers such as Amazon, Google or Microsoft to do too much to help you to use less of their services, so don’t expect any miracles any time soon. They are progressively providing more details on billing and more help understanding this. But reconciling deployments with billing data remains a challenge, and if you want to carry out any meaningful analysis of those thousands of rows of ‘raw’ usage data you still need to wade through treacle.
You can manually download Microsoft Azure resource consumption data as a comma-separated variable file (CSV) and view it in one of the many applications that accept CSV files, such as Microsoft Excel. In response to criticism, Microsoft made the field more intuitive, expanded packed fields and added detail to its CSV data. It used to show resources consumed but not the payment rate or billing amount, and then these details were added – along with a breakdown of daily usage.
Amazon has also responded to customer complaints about the lack of detail in its bills by introducing new reports that include line items for every hour (or partial hour) of AWS activity for an account. For consolidated billing customers, the report displays a line item for all AWS usage by linked accounts and the payer account with two lines – one belongs to the payer and shows the usage charge, the other belongs to the linked account and allocates cost from the payer account charges to the linked account.
Helping with analysis – slightly
You still have to work to analyse this data. It is only available for periods after the date when you opt to get detailed billing reports (which you do on the Billing Preferences page), and it is delivered to a bucket in your Amazon S3 storage account as series of .zip files. Even zip files can soon swell your storage needs, so if you don’t want to pay for the privilege of having them sitting in the cloud you will need to regularly clear them out or download them.
Amazon warns that “they can run to sizes of more than a gigabyte, and may exceed the capacity of tools such as Microsoft Excel to display every line” so you (or your database administrator) may need to import the data into a database. But the billing reports can be helpful: they make blended and unblended rates and costs transparent to customers, show when transitions to different price tiers occur, and highlight the way that Reserved Instance discounts are applied (as Cloud Pro covered here).
You can also organise and track your AWS costs to reflect your business cost centres by using tags. After you have applied these to resources (such as Amazon EC2 instances), AWS generates a Cost Allocation Report as a CSV file with your usage and costs aggregated by your tags and delivers it to your chosen Amazon S3 bucket. Then you can squirt it into your accounting system or other applications that accept CSV files, or exploit Amazon APIs to build your own custom applications.
Improving visibility – slightly
Features such as these can improve your visibility into costs, but how much they help you to monitor, measure, manage, control and analyse these (or understand your bills) depends on many factors. These include: the granularity and frequency of the data provided, whether it’s accurate or an estimate for a monthly charge, the way your IT function and the rest of the organisation deploy and pay for cloud resources, and the tools that are being used to variously collect, collate and analyse cloud usage data.
If you have a complex organisation, large amounts of data to analyse, or want to use an accurate ‘showback’ or ‘chargeback’ cost allocation mechanism (as Cloud Pro explored here) then a spreadsheet may not be up to it. You may want to build your own application, or use a tool such as 2W Insight, Cloud Cruiser, or Uptime Software (and others Cloud Pro considered here) to collect detailed cloud usage information and provide you with dashboard access to real-time costs and billing insights.
None of these are going to stop that programmer from forgetting to turn off a cluster of servers, and they certainly won’t prevent unexpected demand spikes – whether they are caused by oversight, hackers or the unanticipated popularity of applications or services. But at least it’s getting easier to keep tabs on cloud deployments and costs, and to understand who did what, when and how much it cost you, and to minimise the risk that this could make your IT budget swell as much as Easter did your waistline.
Six ways to better monitor and minimise public cloud infrastructure costs:
1. Watch out for changes in pricing and the flexibility of terms and conditions – even if cross-vendor cost comparisons are deliberately made difficult.
2. Take advantage of discounting mechanisms and tools to help you manage these – such as Microsoft Azure Commitment and the AWS Reserved Instances Marketplace.
3. Explore the options for usage alerts and notifications and enable them – whether you are trying to stay in the free usage tier or monitor paid for services.
4. Establish whether the consumption data your vendor provides is actual or an estimate – and factor this into your plans and budgets.
5. Analyse available consumption data – even if you don’t want to pay to use specialist software or build your own, a spreadsheet is better than nothing.
6. Use cloud economics – elasticity suits transient apps but apps that consume the same resources 24/7 may better suit traditional hosting or your data centre.
