Posted by Nick Richardson on Jan 27, 2012.
Errors that only your mobile or tablet customers may see
Every day, more and more of your customers move off their standard desktop computers in favor of smaller and lighter alternatives. Today, many people do much of their day-to-day Internet activities exclusively on their smartphones and tablets. One can no longer assume that the majority of traffic coming to your web application will come from the traditional desktop computer; this is especially relevant for e-commerce sites.
According to data collected by Digby, this holiday season, more than ten percent of Cyber Monday shoppers used their mobile devices to make a purchase, and total mobile purchases accounted for more than 6.5% of all sales made by consumers. Both of these statistics are up significantly from last year, and the growth shows no signs of slowing down. In fact, 67% of consumers reportedly plan to make at least one purchase from a mobile device this year. As this trend continues, considerations must be made for how companies plan and structure their applications, as well the implications on performance and functional testing needs.
Recently, SOASTA was involved in a project for a major retailer preparing to launch a new and highly anticipated product. This retailer estimated that 30% of the launch-time traffic would originate from smartphones and tablets. Because of the unique recording technology built into CloudTest, we were able to construct a test that ultimately exposed several bottlenecks in the mobile order paths, which were not present in the standard desktop browser flows.
Under load, both the tablet and smartphone paths showed a significant number of errors when adding products to a shopping cart:
In the chart above, not only do we see a large amount of HTTP 500 Internal Errors (which occurred whenever the response time for a request exceeded 10 seconds), but there are also a substantial amount of asset requests that return ‘Not found’ during the test (an asset which happened to be vital to the completion of an order). Every one of these errors represented a sale that did not complete. In just 15 minutes, more than 7,000 orders failed due to these errors. To make matters worse, the results described above were produced by a test that was configured to run at only 10% of the volume and concurrent connections expected when the new product launched!
As a result of this test, the customer discovered a problem with their application code that caused these errors and changed it prior to launch.
Errors are bad enough, but what about response times?
This particular test exposed some critical scalability issues that only showed up under load – and only showed up on mobile and tablet devices. Unfortunately for our customer, it wasn’t the only issue observed during this test. Certain page and transaction times were extremely high as well – as shown in the chart below.
Here we see the response times per transaction experienced by mobile users, again at 10% anticipated launch traffic. The final step in the user experience (and in this case, arguably the most important: when the user’s payment is processed) had an average time to complete of more than 30 seconds on the mobile and tablet sites! This is in contrast to the desktop browser test scenarios which showed virtually no slowdown or issues. This helped the customer isolate a specific database contention issue that was unique to the mobile and tablet flows.
Mobile Implications
Fortunately, these issues were detected early, which allowed our customer the ability to address them. If these issues weren’t resolved, a failure of their site would have had a substantial impact on revenue.
Keep in mind, however, that your visitors are utilizing your site for more than direct purchases; they’re also looking for promotions, product descriptions, and locations or directions to brick and mortar stores. This type of usage requires additional design considerations for the layout of your site’s pages. Besides the obvious considerations for the physical differences of the device itself (screen size, touch screen input, etc.) we must also consider bandwidth availability and transfer speeds of mobile devices, and keep those constraints in mind when executing our tests and analyzing results.
Because mobile devices typically take longer to complete a transaction (especially on 3G and Edge networks), the resulting connections from those devices are kept open longer. This has the potential to increase the total number of active connections when user traffic is high. Failure to tune your application and network to properly handle these longer connections may result in missed opportunities. Even if the application servers have ample system resources to handle more traffic, potential customers will be unable to access the application because of insufficient connection availability.
Whether it’s a mobile site or a mobile application, SOASTA CloudTest is able to record directly from mobile devices, allowing anyone to create robust test scenarios without the guesswork involved with user-agent spoofing or other artificial means of deceiving your servers. CloudTest provides the analysis tools required to determine what users would experience when they visit your site, regardless of where they are and what type of device they are using.
Recent Comments