The Performance Beacon

The web performance, analytics, and optimization blog

Is your Enterprise QA organization ready for Mobile?

Unless you’ve been living in a cave for the last 18 months, you should know that Mobile is the way forward to conduct business, deliver media and entertain. There’s still a lot of debate about whether native apps are the right model (according to Marc Andreessen, The application model of the future is the web application model), if HTML5 is finally going to take off or if a hybrid approach represents the best of both worlds. In any case, no one can argue that mobile is just a fad.

At SOASTA, we talk daily with companies trying to figure out their mobile strategy and how they can deliver the best quality apps to their customers. These discussions almost always result in the same conclusion: a majority of companies have been taken off guard by the mobile world and are struggling to transition their QA department to support this new model. It feels like the late ’90s all over again when companies were trying to figure out how to best transition to the Internet era. The mobile era brings somewhat similar challenges, but the requirement to transition is exacerbated by the speed of mobile adoption, proliferation of smartphones and mobile devices, roll-out of 4G and an explosion of social media.

According to The right product

65% of companies are still trying to figure out what is the right product to test their apps. This is due to the fact that, as they figure out their mobile strategy, they realize that the list of requirements for their testing product gets larger. At first they only had to support iOS. Then they realized that Android was also key. Then they wanted a mobile web version or maybe hybrid. They had to think about performance of their front-end, then the mobile app back-end. A lot of testing products specialize in a few requirements and organizations sometimes ends up with 5-6 different products for all their testing needs. This might be a short-term strategy but is not sustainable in the long run and can’t really scale, especially in an enterprise environment where standardization and integration is a must. Many companies that we talk to are very interested in our “platform” approach, allowing them to cover all their requirements with one integrated product.

A new Infrastructure to manage and leverage

52% don’t have access to the required devices. Fragmentation is a critical challenge faced by organizations and one they can’t ignore. But a lot of them are still trying to understand how to manage it. By discussing this with customers, we realize that they don’t invest into devices not so much because they don’t know which one to pick, but because they struggling to integrate them into their development and test life cycle. At SOASTA we’ve been tackling this challenge from Day 1 of the development of our mobile solution, TouchTest. We’ve been building our mobile lab to support the testing of our own product. Today, for each build, thousands of tests are run automatically in our iOS and Android mobile lab. New versions of the mobile apps we use for our testing are deployed automatically on these devices, tests are launched by our continuous integration framework (Jenkins) and test results for each build are available in dashboards along with all of our other tests (unit tests and performance tests). We’re happy today to be able to package these labs and enable customers quickly and affordably.

Skills and processes

According to the World Quality Report, “a majority of organizations characterize their internal teams as ‘average’ at best in their knowledge of core testing processes and methodologies, and not necessarily up to speed with the latest testing tools and technologies” and “less than 5% of firms are fully confident that their testers (internal or external) are ‘best in class’”. Testers have to face the same challenges they had to tackle during the web era: supporting legacy applications running on the desktop (or mainframe) while investing some of their time to transition to the web. Today they have to go through similar transition at a much accelerated pace.

The report also mentions the fact that a third of organizations they’ve talked to lack the testing methodologies and processes necessary to effectively certify mobile applications. Any new technology brings new methodologies and processes, and mobile is not any different. We found this out very early during the development of TouchTest and we’ve been building our own methodology for the last few years now. A couple of weeks ago we decided to share our expertise and methodology by creating a mobile division to help customers set up their own Testing Center of Excellence (TCOE). According to the report, Enterprise companies are more and more interested in industrializing their testing activity and are looking at TCOEs to act as a virtual command center, using a standardized approach and a flexible pool of available resources. According to the report, 60% of companies plan or are developing a TCOE (up from 45% last year). By providing mobile test automation and performance experts, continuous integration test methodologies and management services for internal and external mobile test labs, we hope to be part of their success.


Our own observations and the results of the World Quality Report clearly demonstrate the challenge organizations are facing today. QA needs once again to reinvent itself (and follow the steps of development, which is almost always faster to innovate and to keep up with technology) and find its place into the mobile application lifecycle. In a more and more agile world with shrinking delivery cycles, QA needs to find ways to bring continuous value very early and throughout the whole lifecycle. Organizations should make sure testers have an opportunity to be properly trained to this new mobile landscape.

Additionally, the new mobile infrastructure needs to not only be properly evaluated but also completely integrated within the development and test cycle to bring its full potential. The mobile device is the new desktop, the new server, the new browser. A small difference is that the scale of complexity is twenty-fold: consider the type of OS, version of OS, type of devices, size of devices, the brand, different mobile web browser, different technology for the apps, etc.

Finally, QA organizations need to figure out very quickly if they’d rather invest time and money in dozens of different products or if they want to settle on an integrated testing platform dedicated and built from the ground up for mobile. Testers have a tendency to invest a lot of their time evaluating new products which support just one requirement. We recommend that they take a look at mobile more broadly and understand how their company is going to leverage it in the future. Will there need for testing iOS? Android? Both? Mobile web or hybrid? How about performance or real-time monitoring? Is there a requirement to understand the impact of mobile back-end stress on the front-end of the mobile app? Or a need to run thousands of tests for each build to guarantee a weekly update of the app? These questions are important to answer before investing in any testing products as a bad decision is not only costly but would also reinforce the perception that QA is the bottleneck.

SOASTA Marketing

About the Author

SOASTA Marketing

Follow @CloudTest