The Performance Beacon

The web performance, analytics, and optimization blog

Did Affordability Save the Holidays?

We’ve just emerged from the largest US holiday opener in online history – Black Friday and Cyber Monday 2012! Generally, the web performed favorably. Congrats to the well-prepared!

  • Cyber Monday sales were up 30% from last year, just shy of $2B – $1.98 – from Internet Retailer.
  • Black Friday online sales exceeded $1B for the first time ever according to Reuters.
  • Mobile sales were up 100% amounting to $640M over the two days.

Rounding to whole numbers – that’s $3B in online sales in 2 days!

(Note – Chinese online retailers did $3B in 1 day this month…but that’s another blog.)

The statistic that makes for many proud SOASTANs is that this list from Black Friday’s top performing sites is a who’s who of our customers who prepared with CloudTest! Hooray to them all for legendary performance! Not surprisingly, there were reports that mobile web user experience took a big hit. Might that $640M have been higher with better mobile performance?

Now that “web scale” includes mobile access, testing approaches must embrace the expanded requirement to cover mobile performance. At least for our customers, affordability has had major impact in allowing them to test frequently at realistic scale and include mobile user profiles that were out of reach in the past. It is heartening to see so many companies becoming proactive to avoid “Success Disasters” – those opportunities when all the preparation, marketing, promotion and economy converge for your best possible outcome!



In the five years that SOASTA has been helping customers prepare for success, we’ve seen behavioral change in software testing driven by their end-user activity and affordable access to the resources needed to test earlier and faster, therefore more completely. The lower costs enabled by cloud computing, on a platform that enables anytime access and isolates issues instantly, have evolved best practices that yield results measurable in successes like this year’s holiday opener.

1)    Test early. Many of our customers began a few years back with “Hail Mary” types of tests in production systems sometimes a month or even weeks before Black Friday/Cyber Monday! Once they lived through that drill, they adopted cloud testing as part of iterative development and began ramping up much earlier to in the dev cycle.

2)    Test often. When there are no limits set by high license and hardware costs, testing can become a continuous activity. One customer – on the list mentioned above – had a goal of 68 tests to prepare for their big day. Without cost limits, they were able to run more than 1,100 tests!

3)    Test for more corner cases. When costs of hardware are minimized, the types of tests you can run are limitless. Testers in the past worked around scale limits or tweaked tests to make a 10-user license look like 100. It’s now an everyday practice to run tests of 100% , 200% or even 500%+ user load to stress and measure systems to their max capacity and beyond … quickly. This opens the door to many other test types to stress individual services, tiers, additional apps, and importantly, all the mobile load that could impact success.

4)    Test in Production. Once a taboo, every consumer-facing retailer we know is now testing in production. There is no other way to fine-tune the real prod infrastructure – from firewalls and load balancers to databases. Additionally, the latency effects of global distribution on end users – both web and mobile – can’t accurately be measured anywhere than in the “real” environments.

Affordability makes testing early and often a reality. From the people you hire to do the work, the hardware required to run realistic tests, the cost of the tools, to all the time users save in prep and setup, cost savings across all these areas has led to “Happy Holidays” for SOASTA users!


Brad Johnson

About the Author

Brad Johnson

Brad is a cloud-testing pioneer who joined SOASTA in December 2008. His former roles as head of test and monitoring products at Compuware, Mercury Interactive and Borland prepared him well to disrupt the skeptical and established software quality market with updated approaches and technologies for continuous web and mobile testing.