Cloud Testing Methodology

Best Practices Testing in the Lab and in Production

Despite the attention given to the poor performance and reliability of websites in the last few years, popular websites continue to slow down and even crash, often under expected traffic conditions. What happens now that companies need to respond to the unexpected peaks that come from social networking, external events or promotions, such as the Super Bowl? How can companies accurately test dynamic applications at web-scale, often running in complex infrastructures?

Based on over three years of experience testing from the cloud and leveraging existing best practices and proven performance testing methodologies, SOASTA CloudTest® is the first product to be incorporated into this methodology and extend traditional approaches to address the new opportunities and challenges presented by cloud testing. These include:

• Testing both in the lab and live web-based applications in production
• Leveraging the Cloud to test at both typical and peak traffic levels, from hundreds of users to millions
• Responding to the acceleration of development cycle times by making agile performance testing a realistic alternative
• Generating geographically dispersed load to most accurately represent real-world traffic
• Generating both internal and external load and using both the lab and production environments for the most efficient and effective results
• Analyzing Performance Intelligence in real time to speed problem resolution

The SOASTA Performance Test Methodology is a software QA performance testing strategy with a set of lightweight processes that fit into the software development lifecycle and ensure that online applications are ready for peak usage. The methodology includes testing applications in a lab environment inside of the firewall and processes for testing the production infrastructure used by customers. In addition to testing for baseline performance metrics, the methodology includes failure, duration, spike and break-point testing. It covers standard development life-cycle steps, from requirements gathering through to delivery, with ongoing measurement through iterative cycles.

Testing in the Performance Lab

Cloud testing does not obviate the need or eliminate the benefits of testing in a lab environment as well as the production environment, and it’s important to have continuity between the two. Ongoing performance testing in a lab allows application engineering teams to assess performance over time, and helps catch any show-stopping performance bugs before they reach production. In addition, the lab provides a place to performance test code and configuration changes for performance regression, before releasing changes to production and outside of the normal build cycle. This could include things like a quick bug fix in a page, or a seemingly minor configuration change that could have a performance impact and should be tested before it is deployed. Often, these kinds of changes are deployed with little to no testing and come back later to cause performance issues.

Testing in Production

Testing in production is the best way to get a true picture of capacity and performance in the real world. Testing in production is the only way to ensure that online applications will perform as expected. There are many things that SOASTA’s production testing approach typically catches that cannot be found with traditional test methods. These include:

• Batch jobs that are not present in the lab (log rotations, backups, etc.) or the impact of other online systems affecting performance
• Load balancer performance issues, such as mis-configured algorithm settings
• Network configuration problems such as 100MB settings instead of 1GB on switches and routing problems
• Bandwidth constraints
• Latency between systems inside and outside of application bubbles

SOASTA’s production testing methodology helps identify the invisible walls that show up in architectures after they move out of the lab. Traditionally, testers have been limited to making extrapolations over time about whether small tests on a few servers in a lab can support exponentially higher amounts of load in production. Without proper testing, these types of assumptions always result in hitting unexpected barriers after multiple years of consistent traffic growth. We have seen that successful companies are using production testing to learn things about the performance of their sites that they could have never learned in a lab.