GigaOm Key Criteria for Evaluating Cloud Performance Testing Solutionsv4.0

1. Executive Summary

Cloud computing technologies are relatively mature and have achieved high levels of adoption in many organizations. This adoption requires stakeholders to ensure applications can scale to meet ever-growing demand. Today, these stakeholders include developers, testers, quality assurance (QA) teams, performance engineers, and business analysts. These teams leverage performance testing to ensure an application is performing as expected and users have an optimal experience. Confirming this ability to scale—in terms of users, transactions, and data and processing volumes—is accomplished using performance testing tools.

These performance testing solutions help to isolate issues and identify ways to resolve them. Cloud performance tools are evolving and becoming more robust in their ability to identify problems and bottlenecks earlier in the process. Many have the ability to pinpoint issues and possible solutions. From a database perspective, these solutions help to test and optimize configurations for cache size, bucket size, and input/output (I/O).

This review is focused on cloud-based load-testing solutions in which the ability to scale is based on leveraging hyperscale cloud economics and allowing usage-based billing. This approach expands the ease of use for load testing and removes the need to schedule projects that use limited on-premises testing capacity. Cloud testing enables an application programming interface (API) level of interaction between developers and testing capacity. It also provides automated low-cost ways for operational teams to test applications before and after changes to corporate infrastructure. Such testing reduces the need to have large pools of people online during a change control event to ensure either that their application still works after the change or that a change fixes the issue that initially prompted it.

Business Imperative
Cloud performance testing allows IT organizations to ensure maximum performance from cloud applications in a cost-effective manner. The ability to test performance parameters frequently and cost effectively allows organizations to scale applications and respond to customers’ needs. These solutions enable IT teams to test potentially millions of concurrent users around the world to ensure an application can perform. Cloud performance testing solutions are often sold in a form of “pay as you go,” by which customers are charged for cloud usage for testing only. This model makes it one of the more cost-effective ways to ensure reliability and performance of applications.

Sector Adoption Score
To help executives and decision-makers assess the potential impact and value of a cloud performance testing solution deployment to the business, this GigaOm Key Criteria report provides a structured assessment of the sector across five factors: benefit, maturity, urgency, impact, and effort. By scoring each factor based on how strongly it compels or deters adoption of a cloud performance testing solution, we provide an overall Sector Adoption Score (Figure 1) of 4.4 out of 5, with 5 indicating the strongest possible recommendation to adopt. This indicates that a cloud performance testing solution is a credible candidate for deployment and worthy of thoughtful consideration.

The factors contributing to the Sector Adoption Score for cloud performance testing solutions are explained in more detail in the Sector Brief section that follows.

Figure 1. Sector Adoption Score for Cloud Performance Testing

This is the fourth year that GigaOm has reported on the cloud performance testing space in the context of our Key Criteria and Radar reports. This report builds on our previous analysis and considers how the market has evolved over the last year.

This GigaOm Key Criteria report highlights the capabilities (table stakes, key features, and emerging features) and nonfunctional requirements (business criteria) for selecting an effective cloud performance testing solution. The companion GigaOm Radar report identifies vendors and products that excel in those decision criteria. Together, these reports provide an overview of the market, identify leading cloud performance testing solution offerings, and help decision-makers evaluate these solutions so they can make a more informed investment decision.