When load testing at the HTTP level, it’s important to emulate the browser as closely as possible. In a browser, the HTML page is loaded first. Then, in parallel, the links for the static assets on that page are parsed and requested from the target web server. CloudTest, like most contemporary browsers, defaults to six parallel threads to download the static assets on a page. This provides a realistic method for testing a web application. Within CloudTest, the ability to download assets in parallel is referred to as a “burst”.
Running in sequence would be an alternative to “burst” mode. This means the test application requests the HTML page and, subsequently, all of the static assets are requested in sequence (no parallelism). Unfortunately, this mode only uses one connection, so the test must wait for the response of one request before it sends the next request. What we find is that the time it takes to load a page in a browser is much faster than the time it takes to load a sequenced page. The question is how much faster?
I set up a simple test to show page load times using the CloudTest “burst” method (set to the default of six parallel threads, which is configurable) vs. the “sequenced” method. The test has one user for each method repeatedly hitting the SOASTA homepage. The screenshots below show that by “bursting”, as a browser does, we are generating 4x more traffic per user. In one minute a user hit the home page 82 times when configured with multiple connections, and only 21 times without. The average time to load the burst page was 0.75 seconds, and the sequenced page was 2.85 seconds.
The inaccurate page load times are the most obvious issue. Performance with Sequenced users was 4x slower that Burst users. The not so obvious issue is with throughput. Each virtual user is generating less traffic per second than a real user would. The site is serving fewer HTTP requests per second and lower amounts of bandwidth. Fewer hits per second can be attributed to less CPU usage on the servers. The danger here is that a site could potentially crash with less users than were actually tested. The site might hit a bandwidth limit or a performance bottleneck on CPU in the real world that it never saw in testing.
To show the impact on throughput, I set up a second test using CloudTest Lite. In this test, I have one user hitting the page every five seconds. This should take away any concerns with think time or pacing. Every five seconds we hit the home page. Simple enough.
The charts below report the “Send Rate” (HTTP requests/second) and Bandwidth (Megabits/second). The first chart shows the traffic patterns of a Sequenced user. Since the page takes longer than a second, the HTTP requests are always spread over more than one second. This results in the website never serving more than 23 HTTP requests/second, or 1.6 Megabits/sec. The test is artificially throttling the traffic.
The second chart shows a “Burst” user. The multiple connections used in burst mode allow for a faster and more realistic download of the page. In this example, the website is serving 32 HTTP requests/second, or 2.3 Megabits/sec. This equates to 40% more HTTP requests/second, and 43% more bandwidth per virtual user.
At SOASTA, our methodology with HTTP load testing is to accurately simulate the traffic of a real user. Our goal is to accurately predict how well a site can handle expected traffic. In our experience, doing so is much more than just having the right “Think Times”. Accurate simulation of the way a browser loads a page can have a huge impact on the accuracy of the test. In this simple example we found response times with 4x variability and throughput with 40% variability. When using “sequenced” loads, you could be simulating much lower traffic than you think and you could be overestimating the number of users your site can successfully handle.
About the Author