The Performance Beacon

The web performance, analytics, and optimization blog

It’s spring! Why mobile testing is like fertilizing your lawn

For years, testing has been considered a second-class citizen when it comes to the pecking order of importance in any endeavor. Whether the endeavor is ecommerce related, or any other application, testing has been synonymous with other commodities. The words “I am a tester” might typically get a response along the lines of sympathy card from Hallmark. “Oh, sorry to hear that. Keep plugging away. You’ll make that leap up the hierarchy at some point.”

With today’s complexity found in all areas of software — not just ecommerce, but cloud — and just about anywhere that has software that in some way acts as the front end to some activity initiated by a user, testing has become a complex role that encompasses more than just an assembly line mentality of checking the box and moving onto the next item rolling down the assembly line.

The role of mobile testing across the product lifecycle

Next week at MobileWeek 2015, Connie Quach(@Connie9888), SOASTA VP of Product Management, and I (@DanBoutinGNV) will both speak and discuss the role of mobile testing across the product lifecycle. In particular, I will be addressing:

  • the test coverage model,
  • how it has changed the landscape (pun intended!) in the testing world, and
  • how today’s testing AND development teams must adapt to this changing world to ensure the proper coverage model for their business.

Today’s testing is not your father’s Oldsmobile

Let’s look at the original Oldsmobile first: the desktop.

Desktop application testing

A desktop application test coverage model might have required you to test across two different browsers for the desktop. If you had 300 test cases, you might have had to build 600 tests. Ok, a lot, but not insurmountable from a time perspective. The time might look something like this:

300 test cases x 2 browsers = 600 test cases
600 test cases x 30 mins per test case = 300 hours
300 hours / 6 hours per day = 50 person days
50 person days / 5 people = 10 days of testing

Mobile application testing

So now let’s look at today’s mobile test coverage model. Those same tests today have to allow for more browser types and versions — as well as multiple operating systems with multiple flavors — across multiple device types AND multiple screen sizes and manufacturers. Now look at the test coverage model:

300 test cases x 26 devices = 7800 test cases
7800 test cases x 30 mins per test case = 3900 hrs
3900 hours / 6 hours per day = 650 person days
650 person-days / 65 people = 10 days of testing – 2 weeks
650 person-days / 5 people = 130 days of testing – 26 weeks

And I haven’t even touched on requirements/needs around different data sets, or even coverage analysis around critical path analysis that would be tied back to a test plan based on a firm set of functional requirements of which the application is supposed to satisfy from the original business objectives.

Don’t panic!

But before you panic and start mandating 24-hour work shifts, or go out and hire cruise ships full of crowdsourced testers* or even dream of the nirvana of 100% automated testing, come visit SOASTA at MobileWeek 2015 in New York City next week.

Need more information on SOASTA and MobileWeek? Visit this page
After all, you don’t want your lawn looking like this, do you?

It's Spring image - 2Not a good coverage model, I can assure you. But testing is like fertilizing the lawn. I want all the grass to be dark green, but I want to cover it all with the least amount of effort by me using the drop spreader. And if I don’t cover it correctly, it may not be noticed right away, but like the grass, it’ll show the real results of the coverage soon enough.**

*As an aside, SOASTA ran over 285,000 tests in the first quarter of 2015. How many did your team run? (We did not use 24-hour shifts, cruise ships or crowdsourcing to do it, either).

**In case you’re wondering, I fertilized my lawn three months ago. I live in Florida. Neither my lawn nor my coverage model look anything like either of the lawns in this post.

Try TouchTest for free for 30 days

Dan Boutin

About the Author

Dan Boutin

Dan is the Vice President of Digital Strategy for SOASTA. In this role, Dan is responsible taking the world's first Digital Performance Management (DPM) solution to market as a trusted advisor for SOASTA's strategic customers, and changing the way ecommerce organizations approach the marketplace.

Follow @DanBoutinGNV