The importance of benchmarking clouds

1. Summary

For most businesses, the debate about whether to embrace the cloud is over. It is now a question of tactics — how, when, and what kind? Cloud computing increasingly forms an integral part of enterprise IT strategy, but the wide variation in enterprise requirements ensures plenty of scope for very different cloud services to coexist.

Today’s enterprise cloud deployments will typically be hybridized, with applications and workloads running in a mix of different cloud environments. The rationale for those deployment decisions is based on a number of different considerations, including geography, certification, service level agreements, price, and performance.

Price and performance are often seen as closely – almost inextricably – linked, and this connection is only likely to grow stronger as basic cloud infrastructure becomes increasingly commoditized. For some workloads, cost may be an overriding factor in selecting a cloud provider, and for others, performance is all that matters. For most, a complex combination of these and other factors will lie behind any deployment decision, making it important to ensure that buyers are fully informed about their options.

Frequent high-profile price reductions, equally numerous but less visible infrastructure upgrades, and the less rapidly evolving landscape of certifications, accreditations, and legal considerations combine to ensure that even the best and most informed selection processes require regular review; the best cloud provider for a particular set of requirements today may be surpassed by a competitor tomorrow.

Key findings include:

• There is significant variation in the price and performance of competing cloud solutions, both public and private.
• Lack of consistency in specification or description makes it challenging to accurately compare the capabilities of competing solutions.
• The characteristics of specific applications (web servers, e-commerce applications, Hadoop clusters, etc.) mean that their performance will change from one cloud to another.
• The most effective way to really understand the best place to run a given application is therefore to test how that application performs in different clouds and to build a comprehensive view of the complex balance between price and performance for any given workload.

Feature image courtesy Flickr user Scott Akerman