In May 2013, Australian e-tailer OZSALE.com.au decided to run a FLASH FRENZY event in spite of a competitor’s epic failure during a similar sale. To do so required rolling out an entirely new shopping basket system in a very short timeframe. Previously, the company could monitor its home-grown systems during the course of normal use. However, when your first real test is the biggest event you have ever done, you need assurance that it will stand up to unforeseen loads. This led OZSALE to SOASTA.
FLASH FRENZY generated Click Frenzy Fail
In November 2012, in an effort to reinvigorate retail sales in Australia, e-tailers joined together to participate in Click Frenzy, a 24-hour event where shoppers could snag one-off deals on items ranging from clothing to cameras through the aggregate website. Unfortunately, in what is now known as the Click Frenzy Fail, the website was unavailable for several hours as the event began, leaving shoppers frustrated and unable to procure the bargains they were anticipating.
OZSALE did not want that to be its fate, too, and thus began preparing for an onslaught of traffic during its similar planned event, FLASH FRENZY.
FLASH FRENZY – Preparing for Massive Traffic Spikes
For OZSALE, individual brands are only available on the company’s website for 3-5 days. The company expects surges first thing in the morning and then again mid-day, when new sales start. For the FLASH FRENZY event, however, they had no idea how much traffic would be generated. According to Cliffe, “We expected something with our investment in marketing on TV and the press, as well as massive overspend on Facebook and Google. We needed to find some way of testing before the event and we didn’t want to find out during the event that we had some sort of bottleneck with our site.”
During the testing, most test configurations showed instability along with high CPU utilisation well before reaching 10,000 concurrent users. The OZSALE team learned that a few non-essential API transactions were causing a great deal of stress on servers, regardless of how many users were executing those particular transactions. This caused unstable throughput and premature performance bottlenecking for the rest of the user flows. When test scenarios were run excluding the API requests, server-side utilisation was much lower, response times much more stable, and throughput scaled much better.
This is exactly what OZSALE needed to know.
In the Numbers
During FLASH FRENZY, OZSALE.com.au achieved four times the amount of traffic over their closest flash sale competitors.
- Revenue hit more than $5 million
- As predicted, more than 50% of sales revenue came in via mobile
- 80,000 new members pre-registered prior to the event
- 160% increase in new visitors
- 160% increase in page views and 80% increase in unique browsers
SOASTA Technology at Work
OZSALE uses SOASTA mPulse to understand the impact of application performance on actual user behavior. Using mPulse, OZSALE has access to real-time, actionable intelligence on its web and mobile apps. This enables the company to make adjustments while its user are interacting with its apps, improving their user experience and cultivating customer loyalty.
OZSALE faces the challenge that all retailers face in a new mobile world – ensuring rock-solid quality and reliability for their mobile app. Crashes and slow performance are catastrophic for a flash sale and directly impact bottom-line revenue. The very nature of mobile apps makes them prone to crashes and it is difficult to get feedback on those crashes. Could it be an issue with someone’s phone? Or is it a problem with the app itself?
OZSALE found that when its members first installed their app it would be fine. Yet over time, there would be performance degradation. In a world where fast release cycles are a given, this was a problem that required diligent testing.
OZSALE also needs to detect potential app problems before releasing new versions to their customers on all of the various devices customers use. Using TouchTest, they are able to test their app for every code change, on any number of real devices, while providing continuous feedback to their app developers on the quality and reliability of the app.
“TouchTest was excellent in that regard, really allowing us to record all the different scenarios required for our mobile app. We effectively and continuously run all those scenarios,” says Cliffe. “One test has been going for a couple of weeks, just continuously running our application.”
Where do we go from here?
Today, APAC SALE operate not only in Australia (www.ozsale.com.au and www.buyinvite.com.au) but also in New Zealand (www.nzsale.co.nz and www.buyinvite.co.nz), Singapore (www.singsale.com.sg), Malaysia (www.mysale.my) and Philippines (www.mysale.ph). As the e-tailer expands into Asian markets with their potential for very high traffic loads, it will need to load test every scenario before releasing them into production. “Functional testing and automated recorded tests of every scenario on our site will be important too,” says Cliffe. “TouchTest is vital moving forward. I don’t know if there is any other way to proceed.”
“SOASTA has been extremely easy and effortless to implement. The SOASTA performance engineers that worked on the load test with us were so accommodating for the last minute changes we would make. There were never any issues with time zones even when the team had people from Russia, Singapore and the Philippines and the U.S.,”says Cliffe. “As we get toward the end of the year, which is a naturally busy time, we will use SOASTA to take our load testing to the next level.”
SOASTA is the leader in cloud testing. Its web and mobile test automation and monitoring solutions, CloudTest, TouchTest and mPulse, enable developers, QA professionals and IT operations teams to
test and monitor users with unprecedented speed, scale, precision and visibility. The innovative product set streamlines test creation, automates provisioning and execution, and analyzes real user behavior in real-time to deliver actionable intelligence, faster. With SOASTA, companies have confidence that their applications will perform as designed, delivering quality user experiences every time. SOASTA’s customers are many of today’s most successful brands including Avaya, American Girl, Bonobos, Chegg, Experian, Gilt Groupe, Hallmark, Intuit, Microsoft and Netflix. SOASTA is privately held and headquartered in Mountain View, Calif. For more information about SOASTA, please visit www.soasta.com.