Archive for January 2012

Don’t Neglect Your Mobile Customers

Posted by Nick Richardson on Jan 27, 2012. Leave a Comment

Errors that only your mobile or tablet customers may see

Every day, more and more of your customers move off their standard desktop computers in favor of smaller and lighter alternatives. Today, many people do much of their day-to-day Internet activities exclusively on their smartphones and tablets. One can no longer assume that the majority of traffic coming to your web application will come from the traditional desktop computer; this is especially relevant for e-commerce sites.

According to data collected by Digby, this holiday season, more than ten percent of Cyber Monday shoppers used their mobile devices to make a purchase, and total mobile purchases accounted for more than 6.5% of all sales made by consumers. Both of these statistics are up significantly from last year, and the growth shows no signs of slowing down. In fact, 67% of consumers reportedly plan to make at least one purchase from a mobile device this year. As this trend continues, considerations must be made for how companies plan and structure their applications, as well the implications on performance and functional testing needs.

Recently, SOASTA was involved in a project for a major retailer preparing to launch a new and highly anticipated product. This retailer estimated that 30% of the launch-time traffic would originate from smartphones and tablets. Because of the unique recording technology built into CloudTest, we were able to construct a test that ultimately exposed several bottlenecks in the mobile order paths, which were not present in the standard desktop browser flows.

Under load, both the tablet and smartphone paths showed a significant number of errors when adding products to a shopping cart:

In the chart above, not only do we see a large amount of HTTP 500 Internal Errors (which occurred whenever the response time for a request exceeded 10 seconds), but there are also a substantial amount of asset requests that return ‘Not found’ during the test (an asset which happened to be vital to the completion of an order). Every one of these errors represented a sale that did not complete. In just 15 minutes, more than 7,000 orders failed due to these errors. To make matters worse, the results described above were produced by a test that was configured to run at only 10% of the volume and concurrent connections expected when the new product launched!

As a result of this test, the customer discovered a problem with their application code that caused these errors and changed it prior to launch.

Errors are bad enough, but what about response times?

This particular test exposed some critical scalability issues that only showed up under load – and only showed up on mobile and tablet devices. Unfortunately for our customer, it wasn’t the only issue observed during this test. Certain page and transaction times were extremely high as well – as shown in the chart below.

Here we see the response times per transaction experienced by mobile users, again at 10% anticipated launch traffic. The final step in the user experience (and in this case, arguably the most important: when the user’s payment is processed) had an average time to complete of more than 30 seconds on the mobile and tablet sites! This is in contrast to the desktop browser test scenarios which showed virtually no slowdown or issues. This helped the customer isolate a specific database contention issue that was unique to the mobile and tablet flows.

Mobile Implications

Fortunately, these issues were detected early, which allowed our customer the ability to address them. If these issues weren’t resolved, a failure of their site would have had a substantial impact on revenue.

Keep in mind, however, that your visitors are utilizing your site for more than direct purchases; they’re also looking for promotions, product descriptions, and locations or directions to brick and mortar stores. This type of usage requires additional design considerations for the layout of your site’s pages. Besides the obvious considerations for the physical differences of the device itself (screen size, touch screen input, etc.) we must also consider bandwidth availability and transfer speeds of mobile devices, and keep those constraints in mind when executing our tests and analyzing results.

Because mobile devices typically take longer to complete a transaction (especially on 3G and Edge networks), the resulting connections from those devices are kept open longer. This has the potential to increase the total number of active connections when user traffic is high. Failure to tune your application and network to properly handle these longer connections may result in missed opportunities. Even if the application servers have ample system resources to handle more traffic, potential customers will be unable to access the application because of insufficient connection availability.

Whether it’s a mobile site or a mobile application, SOASTA CloudTest is able to record directly from mobile devices, allowing anyone to create robust test scenarios without the guesswork involved with user-agent spoofing or other artificial means of deceiving your servers. CloudTest provides the analysis tools required to determine what users would experience when they visit your site, regardless of where they are and what type of device they are using.

The Importance of “Bursting” Static Assets

Posted by Ed Salazar on Jan 19, 2012. Leave a Comment

When load testing at the HTTP level, it’s important to emulate the browser as closely as possible. In a browser, the HTML page is loaded first. Then, in parallel, the links for the static assets on that page are parsed and requested from the target web server. CloudTest, like most contemporary browsers, defaults to six parallel threads to download the static assets on a page. This provides a realistic method for testing a web application. Within CloudTest, the ability to download assets in parallel is referred to as a “burst”.

Running in sequence would be an alternative to “burst” mode. This means the test application requests the HTML page and, subsequently, all of the static assets are requested in sequence (no parallelism). Unfortunately, this mode only uses one connection, so the test must wait for the response of one request before it sends the next request. What we find is that the time it takes to load a page in a browser is much faster than the time it takes to load a sequenced page. The question is how much faster?

I set up a simple test to show page load times using the CloudTest “burst” method (set to the default of six parallel threads, which is configurable) vs. the “sequenced” method. The test has one user for each method repeatedly hitting the SOASTA homepage. The screenshots below show that by “bursting”, as a browser does, we are generating 4x more traffic per user. In one minute a user hit the home page 82 times when configured with multiple connections, and only 21 times without. The average time to load the burst page was 0.75 seconds, and the sequenced page was 2.85 seconds.

The inaccurate page load times are the most obvious issue. Performance with Sequenced users was 4x slower that Burst users. The not so obvious issue is with throughput. Each virtual user is generating less traffic per second than a real user would. The site is serving fewer HTTP requests per second and lower amounts of bandwidth. Fewer hits per second can be attributed to less CPU usage on the servers. The danger here is that a site could potentially crash with less users than were actually tested. The site might hit a bandwidth limit or a performance bottleneck on CPU in the real world that it never saw in testing.

To show the impact on throughput, I set up a second test using CloudTest Lite. In this test, I have one user hitting the page every five seconds. This should take away any concerns with think time or pacing. Every five seconds we hit the home page. Simple enough.

The charts below report the “Send Rate” (HTTP requests/second) and Bandwidth (Megabits/second). The first chart shows the traffic patterns of a Sequenced user. Since the page takes longer than a second, the HTTP requests are always spread over more than one second. This results in the website never serving more than 23 HTTP requests/second, or 1.6 Megabits/sec. The test is artificially throttling the traffic.

The second chart shows a “Burst” user. The multiple connections used in burst mode allow for a faster and more realistic download of the page. In this example, the website is serving 32 HTTP requests/second, or 2.3 Megabits/sec. This equates to 40% more HTTP requests/second, and 43% more bandwidth per virtual user.

At SOASTA, our methodology with HTTP load testing is to accurately simulate the traffic of a real user. Our goal is to accurately predict how well a site can handle expected traffic. In our experience, doing so is much more than just having the right “Think Times”. Accurate simulation of the way a browser loads a page can have a huge impact on the accuracy of the test. In this simple example we found response times with 4x variability and throughput with 40% variability. When using “sequenced” loads, you could be simulating much lower traffic than you think and you could be overestimating the number of users your site can successfully handle.

 

“Minority Report” Defining Today’s User Interface

Posted by Tom Lounibos on Jan 18, 2012. Leave a Comment

Think back to Steven Spielberg’s 2002 classic, “Minority Report”, the movie that explored the future of human/machine interface in 2054. It was both cool and exciting, and it was also years away from reality. Or was it? Here we are, now ten years removed, and some of these concepts, gesture-based applications (everywhere), hovering gestures (Apple patent), 3D navigation (everywhere), and voice (Siri), are very much real.

John Underkoffler is the UI designer who built the technology for Minority Report and in real life, too. His 15-minute TED presentation is well worth watching.

From logistics and supply chain to financial services, new applications evolve every single day to leverage advancements in man and machine user interfaces. In many cases there is simply no other option than to rapidly wade through the enormous amounts of data these applications generate in search of the actionable intelligence.

One of these Big Data issues occurs while performance testing web and mobile applications. These applications simulate millions of people around the globe hitting a web site or mobile application to provide predictive analysis as to where performance bottlenecks may occur. In doing so, they generate terabytes of performance-related, real-time data. Analysis of this data is extremely difficult using traditional, one-dimensional test tools because problems are often buried amongst layers and layers of data.

Over the past few years, companies such as Splunk and SOASTA have emerged because their multi-dimensional capabilities enable testers to see several pieces of performance data on the same timeline and in a single view. These kinds of user interfaces allow for real-time resolution to problems such as performance for a growing number of leading companies.

Of course new problems arise as these problems are solved. The next generation will surely struggle with delivery of a stable user experience when application UIs are gesture-based, voice-based or even 3D. I will leave that question for another day. The bottom line is this: Minority Report has arrived. Is your team ready?

 

Finding Bottlenecks Through Monitoring and Iterative Load Tests

Posted by Yuriy Verkhovoy on Jan 05, 2012. Leave a Comment

A medium-size web development shop engaged us for its first experience with external, scale testing on a brand new application scheduled to roll out a few months after we started. The application is written in Ruby on Rails (ROR) with a MySQL backend, and serves as an internal social networking hub for a company. Through this application, users can create projects and tasks that are both work related and personal, invite others to join them in completing these tasks, share their calendars, and exchange instant messages with other online members. The front end is entirely JavaScript driven and consists of a number of widgets, all of which are updated through AJAX calls to the server. In addition, the client intends to host this application in the cloud, which would be a first for them.

The test consisted of 10 scenarios including simple browsing of the site, creating projects and tasks, and randomly sending instant messages to other online members. The most challenging part of test creation was dealing with the asynchronous nature of the application. There were no clear-cut events that broke the script into URLs or pages. There was only one URL followed by a number of AJAX calls to the server. CloudTest’s nested transaction grouping allowed us to put these requests into logical groups that represented all of the main business processes to be tested.

The first test was against a single server that contained all of the moving parts of the application and was hosted onsite. The application performed well until we reached 200 simultaneous virtual users. At that point, things got really slow. At about 300 users, the site was unusable and every request came back with a HTTP 500 error.

Our second test was against a more distributed environment. The application server and the database were split between two different machines, which were hosted in Amazon EC2. The site did slightly better, topping out at 300 users. Using CloudTest’s monitoring, we noticed that the database was not working very hard and that the web server was at nearly 100% CPU utilization for most of the test.

During subsequent tests we tried a variety of configurations. CloudTest makes it very easy to do iterative load testing. We spread the load even more by putting several application servers behind an EC2 load balancer and increasing the horsepower of the database server. Although most of these changes made a visible improvement, we were still faced with an obvious bottleneck. At roughly 800 simultaneous users, the application started to slow down, and at 1200 users it became almost unusable. At this point all of the usual suspects looked innocent. The app servers’ CPU never got higher than 50%, the database had plenty of breathing room and we were not limited by bandwidth.

When we looked at the “Collection Summary” report from CloudTest we realized that one particular transaction, which was taking longer to complete than others, needed more attention during the test. As you can see from the chart below, the “CREATE TASK” transactions took significantly more time to complete, even when there was little load on the system.

The ops team monitored the app server with New Relic, which has a ROR profiling module and is integrated into the SOASTA dashboards. We found that a CREATE TASK business process was using a ROR ORM Engine called Active Record to perform MySQL queries. A particular query in the CREATE TASK process was not “tuned”, and instead of running one or two SQL statements to retrieve the data, hundreds of queries were run each time a list of tasks was requested by the application. Although the queries themselves were very lightweight, running hundreds of them in sequence for each process slowed things down considerably.

Our findings were brought to the attention of the development team. The team was able to revise the process to access the same data with just one query to the database. As a result, in our subsequent tests we saw a great improvement in the CREATE TASK transaction. Monitoring the infrastructure during a load test once again proved invaluable, and having the results on the same timeline helped surface the issues much more quickly.

 

Email Us!
Subscribe to our Feed!
Find us on Facebook
Follow our Tweets
See our pics