The Fragility of Web Applications
Most web applications have some code in them that if utilized even slightly more than expected could send the whole stack toppling over. Web apps and their associated infrastructure are fragile and they must be performance tested thoroughly across all key functional areas. This testing should be done above expected traffic levels… the following is an example of why this is so important.
I was recently testing for a very large online retailer. Their site has the typical shop, buy, and self service functional areas. On this particular application there is an Ajax call to create an empty shopping cart (sometimes referred to as a transient session) as soon as you start browsing the catalog. It’s a lightweight and seemingly harmless GET request that passes through to the app tier to initialize the empty shopping cart. This cart is an in-memory object at the application tier.
What we discovered through testing was that if we generated the exact profile of traffic they were expecting with people browsing the catalog and creating empty shopping carts, along with customers adding products to the cart, then there was enough capacity to perform well. However, if we adjusted the load mix just slightly to have either more empty carts, or more carts with items in them, then the entire application slowed down and ultimately fell completely over. This affected not only people in the shopping experience but everywhere… the entire site went down.
This really got me thinking about how fragile apps really are unless you test all of the different components past their expected load levels, and assess not only their performance, but also the performance of the components around them.
Every web application has a weak link somewhere. Do you know where yours is? I bet that a very small load test that makes one particular type of request directed at your application could have catastrophic results. It could be 10% more users logging in than normal, or the worst case, more people trying to check out than you had planed for. I’m amazed at how many people market a flash sale and don’t change anything on their application to account for a totally different load profile than normal. We need to find those weak links and build them out to be more resilient.
Dan,
Well said. I know quite a lot about pairwise and other more sophisticated and thorough applied statistics-based test design approaches. When I do talk to performance testing experts I respect a lot, they almost always tell me performance testing / stress testing / load testing don’t really need to focus much on different types of combinations of, e.g., hardware configuration options, software configurations, the amount of information cached, etc.
Among other implications of that common way of thinking (if I could accurately call it that), that view implies that keeping the percentage of empty carts constant across tests would be perfectly fine.
Your example clearly calls into question that common way of thinking.
IF it were desirable to do so in performance testing scenarios, it would be easy for test designers to create tests that maximize variation between tests. These tests would very efficiently identify potentially troublesome combinations. That’s a big if; to be clear, my mind is still not made up on this point in the performance testing space because I don’t have enough experience in the area or access to solid data on the topic. Most experts I ask say it is not worth the trouble to create highly varied scenarios in typical performance tests. Your example, though, is a reminder that some test designers might be too quick to discount the value of ensuring variation between tests.
- Justin (the admittedly-potentially-biased founder of a combinatorial test case generating tool called Hexawise)
Trackback: http://www.real-user-monitoring.com/why-application-performance-management-helps-you-win-with-your-application-launch/