The Performance Beacon

The web performance, analytics, and optimization blog

Randomizing Click Paths Inside of a Test Case – Not a Good Thing

QA and performance test cases are usually a sequential set of steps to take action in an application.  Something like this:

Step 1: Home page
Step 2: Click catalog
Step 3: Click ‘clothing’
Step 4: Choose the ‘fancy sweater’
{some other steps in the checkout process abbreviated for space}
Step 13: Click place order

The key here is that the test case is a structured repeatable set of steps… more on why this is important later.  There have been times when I have been presented with a test case that looks something like this:

Step 1: Home page
From here:
70% of the users click on catalog
30% click on login

Catalog subflow:

15% click ‘Clothing’
11% click ‘New arrivals’

Clothing subflow:
New arrivals subflow:
Login subflow:

This test case is actually not very useful in nearly any situation you could imagine using it, and certainly not in the most common applications for performance testing: baseline tests, stress tests, problem isolation tests, throughput tests… you get the idea.  In addition, it’s complicated to write, maintain, and offers little flexibility in troubleshooting and load calibration.

I never understood where test cases like this came from until I recently watched a live demo of one of the world’s leading web analytics tools.  This particular tool visualized user click paths by starting with a vertical row on the left of all of the landing pages on a web site.  Usually, the majority of users were landing on the home page, but sometimes it was a microsite or a specific popular product where a bulk of users entered the site and then navigated on from there.  When you clicked ‘home page’, for instance, the tool branched out showing percentages of user traffic that went on from the home page to other pages.  It was at that moment when I realized where these test case designs were coming from: analytics tools!  Unfortunately what comes out of an analytics tool doesn’t necessarily translate into a useful test case.  The analytics tool provides you with workload data and traffic numbers, first and foremost.  Secondly, it tells you where users are spending time on the site… and from there you should take that information and design modular and reusable performance scenarios.  To make matters worse, there isn’t a testing tool on the planet that makes writing test cases like this easy (because they were never meant to exist like that).  A die-hard QA engineer will most likely tell you that this is not a real test-case as defined anyway, so it makes sense that tools wouldn’t make it easy to do.  Yet I do see them out there a lot.

Test cases in performance and QA are meant to be steps of actions that can be used to generate a certain load level or reproduce a problem or error condition.  They are meant to be followed closely and at the end have a very specific outcome, assuming certain conditions.  These outcomes can be as simple as placing an order and getting a success page or as specific as a certain click path that generates a specific error.  In performance, the goal of a workload mix is to generate a specific amount of load.  This load is usually in the form of orders per second, page views per hour, and so on.  When problems are encountered in any form of QA the test cases are boiled down to just those needed to isolate and reproduce a problem.  Test cases that are comprised of randomized click paths make all of those things extremely difficult.  Don’t vary the percentages of users executing test cases inside of a running load test.  I have never found an application for this kind of test in 12 years of running tests for the biggest online applications*.  (*this statement limited to the web space)

Designing your test cases like this is attempting to blend two sciences together: workload modeling and test case development.  Don’t mix the two together.  A better way to approach this is to have modular test cases and a workload model built using the right mix of those modular test cases.  Control the execution of those test cases at the Composition level.   Object oriented programming teaches us to separate functionality into the object in which it belongs.  The same applies here.  Another concept that applies is that of the ‘business process’.  Many monitoring technologies and engineering methodologies are based on the concept of a business process.  A BP is a simple set of transaction steps that results in a specific action at the end (order placement, logging in, modifying account info, etc).  There is sometimes a little overlap in shared actions across business processes to account for in the workload, but it’s usually minimal (logging in for instance can happen in a few paths throughout the site).

Here is a summary of the points raised:

Why having randomized click paths in a scenario is bad

1) When you run this test multiple times you’re going to get different results (close at best statistically, but more often than not completely far off in the real world).  This makes calibration very challenging and you can never get it the same consistently and repeatedly.
2) If you hit a problem with one area of the site you have to rewrite, duplicate, or drastically change your test cases to isolate the problem and reproduce it.
3) Maintenance is hard – test cases like this become animals.  You actually have to change your test case to increase the amount of load going to a certain area of the site.  Not good.  Test tools give you mechanisms for controlling this logic outside of the test case for a good reason.

Why having modular, independently controllable scenarios is good

1) You can tweak the test load to a solid calibration.  I recently calibrated a site to within .01 of the targets on all page views at the end of a test because of modular scenarios that could be tweaked to achieve very specific page view targets.  This would be impossible to do reliably with random click path scenarios.
2) If you need to stress an area of the application weeding out the scenarios you don’t need is easy and adding load to one particular area of the site can be achieved by just cranking up the virtual users for that one scenario.  This, as opposed to cutting part of the script out, creating a new one out of it… etc. etc.
3) You can adapt to changes in load by simply modifying the workload, and to changes in test cases by modifying the test cases.  One shouldn’t need to rewrite a test case because recent analytics data shows that more people are creating accounts than logging in.  Simply changing the virtual user count to increase load on a registration checkout scenario is so much easier.

Functionality needs to be abstracted out to the highest level possible that still allows you to achieve business process and transaction goals. Configure the load levels, percentages, and ramp ups in the test definition and not the test case.

SOASTA Marketing

About the Author

SOASTA Marketing

Follow @CloudTest