When SOASTA first burst onto the scene, performance testing was considerably different.
Traditionally, testing was done after a website had been completely created; testing during production, as SOASTA’s Ed Salazar puts it, was “a no-no.” However, with its revolutionary performance testing tools, SOASTA looked to change this perspective.
Testing in production: Real world examples
Salazar, CloudTest product manager and senior performance engineer, noted that the main component that set the company’s testing tools apart from the beginning was its ability to provide real-time analytics. This initially spurred engineers to look at performance testing in a new way.
“We had a product that had the ability to do real-time analytics so we were interested in getting people to test in production,” Salazar said. “We didn’t have a fear of testing because we had the ability to see the data as it came in really quickly … and at the same time we had the cloud at our backs so we didn’t have a worry about getting all the servers needed to run these high scale tests.”
Some of the first big clients to approach testing in this way with SOASTA were Hallmark and Intuit. Both groups were getting prepared for large spikes in client traffic, due to upcoming holidays and tax time. Salazar noted that both Hallmark and Intuit had considerable problems in connection with user surges: Hallmark’s website had crashed two years in a row around Valentine’s Day and Intuit’s TurboTax crashed one year prior as well around the tax due date. Both companies knew that if they didn’t address the issue, they were likely to lose significant business in the upcoming season. As a result, Hallmark and Intuit decided to test in production, which was unheard of at the time.
“They came to us with their hair on fire,” Salazar said. “They had no other choice.”
While the situation mandated testing in production, both companies came out of the process on top. Thanks to SOASTA testing solutions leveraged during production as opposed to after seasonal adjustments were complete, Intuit didn’t crash despite the influx of users during tax season, and Hallmark’s website did not go down during Valentine’s Day for the first time in three years.
Seasonal preparedness: An ongoing process
When it comes to getting a website ready for seasonal traffic spikes, Salazar pointed out that it is an ongoing process. Some organizations will look to begin their preparedness efforts only a few short weeks before they expect season surges. However, this does not give them enough time to carry out all the performance and load testing necessary.
“Seasonal readiness testing doesn’t start a month before Black Friday,” Salazar said. “People start thinking about seasonal readiness in April and they’re getting ready for an event bigger than last year. They need to start earlier, especially if they have to handle millions of transactions, or upgrade infrastructure.”
In addition to starting the preparedness process much earlier in the year, Salazar noted that companies are now looking to test not only the back end infrastructure, but the front end features as well, bringing the end-user experience to the forefront.
“So it’s no longer a situation where people are saying ‘Well, I need to do load testing so I can make sure that 100,000 people can hit my site simultaneously,'” Salazar explained. “Now, it’s ‘Well, I need the 100,000, but I also need them to come in within three seconds because if they don’t, people will bounce off of my site and go somewhere else.”
In this way, it’s not only the capacity of the website that needs to be tested and taken into consideration with seasonal initiatives, but the performance of the platform as well. This brings a number of other components into the equation which must be factored in to ensure the complete readiness of the site for the season. Salazar pointed out that this type of environment makes things much more difficult for companies looking to ready their websites for seasonal traffic spikes: Either they can do a minimal job of testing the myriad features and components and potentially miss certain performance issues, or they can utilize a real user measurement system.
Real user measurement: The buzz of the industry
Salazar said that it was just these conditions that spurred the creation of mPulse, SOASTA’s Performance Testing system. With a RUM solution, companies can see in real time how many clients are interacting with their website, their overall user experience, as well as their conversion and bounce rates. On top of these beneficial capabilities, since these users are already using the site, the data collected is free.
“With RUM you know exactly what your customers are doing,” Salazar explained. “You know exactly what they did last year, you know exactly what they’ve been doing for the last two months, you know exactly what they did yesterday. So you pattern your load tests after the real traffic in the real world.”
A RUM system takes the guesswork out of testing. However, Salazar noted that this can only happen when the team in charge of the front end effectively communicates with the back end team. With this type of data in hand in connection with improved collaboration, seasonal preparedness testing can be carried out much more efficiently and accurately.
“Load testing has gotten better because monitoring has gotten better. The front end guys know what customers are doing, they just need to pass that info on to the back end guys. They can then rely on the back end guys to make sure the site stands up.” Salazar said.
Overall, a RUM system allows both teams to work together to effectively prepare a website for user surges. Instead of guessing how many individuals may visit the site, front and back end teams can simply consult the mPulse to see historically how many users they had at a certain time, what their current traffic patterns look like and ready the site accordingly.
Such an approach can also help break down the silos that can appear between front and back end teams.
“Let’s not just say ‘Hey, I’m a back end engineer, browser rendering is not my problem…let the Dev team deal with that,'” Salazar noted. “We can’t continue to do things like that…that’s not going to make anybody better.”
By working together and sharing data across the different teams, both can have all the information needed to test and ensure optimal performance, allowing staff members to carry out their tasks much more efficiently.
Salazar advised that dev/ops teams leverage both RUM and load testing to prepare for seasonal spikes, as the two approaches provide the halves of a complete, whole solution.
“If you think about load testing versus RUM data, RUM data is sort of reactive because it’s happening in real time,” he explained. “Load testing has to be proactive, we have to think about all the crazy things customers might do and then we have to stress the environment in each one of those ways to figure out if we can find a new bottleneck.”
RUM and mobile: Changing the way traffic hits the website
SOASTA’s real user measurement system has also helped businesses prepare for the mobile revolution. Salazar noted that the rapid adoption of mobile devices among users has shifted how these individuals connect with websites.
“The way traffic hits our sites today has changed dramatically because everybody has a browser in their pocket,” Salazar said. “When you have users that are changing the way traffic hits the site, you have to accommodate for that somehow.”
This is where RUM again comes into play. Salazar explained that the industry is moving toward a “world of customization,” especially when it comes to tagging items on the mobile application. In order to allow for real user measurement on mobile devices, IT teams have to custom tag each feature on the application so they can see each gesture as well as how long these actions last. In this way, mPulse stands apart from other RUM systems because it is one of the only ones on the market to enable this kind of custom tagging.
The mobile revolution has also resulted in a considerable shift in load testing. Salazar said that a number of retailers noticed that when users visited their regular website from a mobile browser, it took much more time – three times longer, in many cases – to load. However, SOASTA’s CloudTest system was specifically designed to address this issue.
“What we’ve done in CloudTest is to make sure we can record directly from the device and build new test cases on the mobile device,” Salazar said. “I need to be able to record [mobile device traffic] and not just guess. Using CloudTest, you can actually record and see it yourself, you know exactly what’s coming off of it because you tracked all the HTTP messages.”
While SOASTA boasts a number of solutions specifically geared toward mobile, Salazar also noted that this is a new arena for load and performance testing and the environment is continually developing.
“Mobile is definitely changing things, but one thing to remember is it’s sort of the Wild Wild West still on how to do user experience on mobile devices,” Salazar said.
There are a range of new considerations to make when testing for mobile, including the users’ data plans and how information is collected. However, the industry has come a long way in terms of its load and performance testing capabilities on the mobile platform.
The value of SOASTA solutions
All told, SOASTA stands apart from its competitors because it provides clients with the top three components they are seeking: speed, scale and actionable intelligence. Salazar pointed out that of the three, actionable intelligence is the main factor to consider with testing processes. Because the website must be functional before testing can take place, the data in connection with the website must be collected quicker, which thus allows for faster performance testing.
Salazar explained the time is of the essence in many situations, as in both native and agile development projects, the product can suffer if the team runs out of time and can’t test all the aspects it would like to. For this reason, SOASTA’s professional services have become critical. With this offer, companies can leverage SOASTA’s professional engineers and testers to script, run and provide a testing plan while the group’s internal team takes care of functionality tests. The business can even have their employees watch the process so that they can bring the tools used in-house afterwards.
Overall, SOASTA services are akin to a race car and its driver: The load and testing tools offered are the best in the industry, and clients can leverage professionals to utilize the products in their environment, creating a winning combination.
About the Author