The Performance Beacon

The web performance, analytics, and optimization blog

How to test smarter with RUM-based performance testing

How to test smarter with RUM-based performance testing

I started my career as a software test engineer. I still remember the last few days before my first big release — wracking my brain for any potential missed test cases. So stressful.

Though my role and the use cases have changed, I still do this today. In the performance world, we wonder things like “Did we test for enough load? Did we test all the critical paths? Did we test the session paths most traveled on our site?  What about different locations? Networks? Devices?”

Engineering organizations have long been searching for confident answers to these questions. We all have to test, so it seems like there should be a smarter way to do it. There have been some pursuits with marketing analytics or server logs, but the holy grail of that accurate, precise, and – most important – complete test plan has remained elusive… until now.

Introducing Smart Testing

Smart Testing is a feature that is unique to SOASTA. That’s because we’re the only performance solution that combines testing with monitoring and deep analytics – and because we’re the only company that lets you collect and keep 100% of your user data.

Using your mPulse Real User Monitoring (RUM) data and mPulse’s Data Science analytics capabilies – combined with our years of experience and proven testing methodology – you can now create complete, statistically accurate test plans for your web applications or sites.

As already mentioned, we are the only solution that empowers you to do this because we are the only RUM solution that gathers and keeps all of your user data. Here’s why this matters:

Why you need real user data

Real users don’t behave in predictable ways. With RUM data, you can see actual session paths, bounce rates per page, think times between clicks, and geographically where your users are browsing from. (That’s just a start. Our mPulse RUM beacon gathers dozens and dozens of other user metrics.)

Why you need to keep all your data – forever

With mPulse, you gather all your data – not just samples – and you keep it all. This lets you run analysis on it for previous critical periods, months, and even years in the past. You can aggregate all these critical datasets into a prediction of what you need to test for to ensure you’re covered. Otherwise you’re guessing. And when you’re guessing, you’re not TESTING – you’re “testing”.

Performance "testing" versus performance testing

Video > Welcome to the Intelligent Testing Revolution

How Smart Testing works

So what happens when you stop guessing? SOASTA Smart Testing automatically generates a full test plan report, which will tell you exactly what to test and how to test it. You can determine your peak second, minute, hour, and day (surprise, they’re probably on different days!) and derive the volumes you should be testing for each page group and session paths.

1. Determine the pages and paths you need to test

A look through your most popular session paths allows our data science to determine the pages and paths that you need to be testing for optimal coverage, including the think times for each page, or how long a user stays on a page before they click to the next one.

Page group distribution during peak hour
Page group distribution during peak hour

In addition, we call out bounce paths – that is, which pages will see more traffic but that users will leave after they visit. This is also good insight on where you might need to focus your optimization efforts.

Lastly we make sure to specifically cover the paths that most lead to conversion.

2. Learn how to distribute traffic through sessions

Once you have the sessions, you need to know how to distribute traffic through them — from where and at what rate.

Session path analysis with think times
Session path analysis with think times

3. Calculate number of virtual users and duration of ramp time

Choose your preferred cloud provider for load generators and Smart Testing will let you know exactly how many virtual users should come from each location and how long your ramp time should be to the full load.

Test distribution and ramp chart
Test distribution and ramp chart
See an example > Smart Testing test plan for


SOASTA Smart Testing lets you:

  • be more efficient and effective with your testing,
  • reduce risk and cost, and
  • produce a higher quality site and more revenue – the true holy grail of a digital business!
Case study > Office Depot’s digital transformation journey: From back-room QA lab to digital performance management
Click to learn more about RUM-based performance testing

Lauren Younger

About the Author

Lauren Younger

Ever since her first TI-99 when she was four years old, Lauren has been interested in computers and software. Her career started as a software design engineer in Test, until she realized that the best way to prevent bugs was to not design them in – and thus made the leap to product management. Today, as Director of Product Management at SOASTA – where she's responsible for mPulse – Lauren loves that her role brings together her love of testing and analytics, as well as her keen interest in everything that makes the internet run. Fun fact about Lauren: The first product she owned as a tester was Clippy from Microsoft Office!

Follow @llyounger