Alexandre Dumas’s classic portrayal of a young man seeking to join the elite guard of his day seems an unlikely source of inspiration for a blog post about digital performance management, but there’s something about groups of three — each balancing the others’ strengths and weaknesses — that makes a great team.
And, while most operators today might view digital systems monitoring in terms of two players — synthetic and real users — there is a third member of the team that turns performance monitoring into performance management: the virtual user.
What does each user agent represent?
To understand how real, synthetic, and virtual user agents complement each other, it’s critical to understand what each represents. Sorting out why one might need both synthetic users and virtual users, for example — or even what the difference between the two might be — can be quite confusing.
Here’s a useful guide to the three user types.
Monitoring the actual experiences of users running in all of the different network, platform, device and geographical environments that your site or application will experience is absolutely essential to creating a near optimal user experience each and every time.
Real users allow you to measure what the end user actually experiences through their front end stack (browser or app, OS, etc), but at the cost of being able to control the conditions in which the user is running those stacks. Being able to capture anomalies quickly, especially for things like key transactions or usage under exceptional loads requires the ability to collect data with more known initial conditions or baselines.
The easiest way to run a known transaction end to end from the user device to backend services with known conditions is to use an artificial “user” to drive that transaction. Synthetic user monitoring simplifies asking questions like “is the backend behaving the way it should at this moment for this transaction” or “are users on the east coast of the US experiencing worse performance than users in Europe?”
Unfortunately, because these systems don’t scale well, and these synthetic users can themselves affect performance if they generate too much load, these users don’t scale well at all. So, how do you create a baseline of expected behavior on the backend systems that have to scale to meet customer demand when stressed?
Virtual users aren’t meant to be used as day-to-day real time monitoring of current conditions, but are intended more to give a solid baseline measurement of backend performance under varying amounts of load. Those baselines can be regularly verified in production, and operators can be warned if virtual user performance slips out of expected ranges.
Virtual users can also be run at incredible scales during scheduled load and stress tests. Their initial conditions can be closely controlled, though the combination of scale and control of conditions comes at the cost of driving load primarily against the backend, without driving actual browsers or mobile applications.
How virtual users complement real and synthetic users
So if those are the three musketeers, and we have a pretty good idea of how synthetic and real user monitoring complement each other let’s explore what virtual users bring to digital performance management.
As our CEO, Tom Lounibos, is wont to say, cloud testing introduced the concept of load testing actual production systems, instead of focusing only on “staging” environments before release to production. This is a powerful concept for today’s rapidly evolving digital applications, but it’s only possible if the load testing system is capable of dynamically adjusting load from an extremely light baseline to a massive peak load without rebuilding the load tests.
The way SOASTA achieves this is by utilizing CloudTest to drive virtual users through the system at a load that can be adjusted throughout a test run as needed. So you can choose to either run “baseline” tests as separate tests from time to time, or continuously run a load test that can be throttled up and down as needed over time.
Low levels of testing give you a baseline of behavior for these test users, which can be compared to behavior when the throttle is turned up. (Low level tests are also great for finding code changes that negatively affect performance or break the load test scripts before a critical load test finds them first.)
This comes at the cost of not using actual browser and device profiles, of course. You won’t know from a virtual user if the experience is better in Chrome or IE, for instance, from a virtual user in a load test. However, you will be able to confirm that your backend servers can handle any anticipated (and all but the most unimaginable unanticipated) loads when they come.
How real and synthetic users complement virtual users
Equally important to the information that virtual users brings to understanding digital performance is the power of utilizing real and synthetic user measurements when designing virtual user load tests. Beyond the simple comparisons of all three types of data — which alone can help greatly with troubleshooting performance and availability problems — there is the power of using real and synthetic data to shape the tests you run.
Example 1: Real user monitoring data will tell you a lot about the situation that cost you the most in terms of whatever your conversion metric might be
Sometimes the situations that cost you the most aren’t obvious. We’ve seen situation in which specific product pages had serious conversion issues, though not all such pages did. Real user data can make that clear, and then load tests can be created to test that condition, at least until you’ve confirmed the problem no longer exists. (Even then you can run the test occasionally as a form of regression testing.)
Example 2: Using the virtual user baselines noted above as additional baselines for real and synthetic user measurements
Does your backend perform differently under varying loads? When is a real or synthetic measurement truly an anomaly vs expected (though not necessarily “acceptable”) behavior? Having a strong, controlled profile of your system performance gives you greater confidence in tagging events as unexpected versus expected.
The most complete protection against performance-related failure needs a three-user team with:
- synthetic users giving you a sense of user experience in well regulated “experiments” across browsers and devices,
- real user monitoring telling you about the actual experiences discovered by actual users, and
- virtual users enabling you to build a feedback loop to test, measure, and verify performance variability and resiliency.
With this combination, you have the makings of a complete picture of where, how, and why performance affects user outcomes.
And understanding user outcomes enables understanding both business and technical outcomes.
- Performance Monitoring 101: A beginner’s guide to understanding synthetic and real user monitoring
- 5 reasons why Digital Performance Management should be your New Year’s resolution
- How much load can your site handle? A concise guide to knowing what, when, and how to load test
About the Author
James Urquhart is Senior Vice President, Performance Analytics at SOASTA, Inc. Named one of the ten most influential people in cloud computing by both the MIT Technology Review and the Huffington Post, and a former contributing author to GigaOm and CNET, James brings a deep understanding of disruptive technologies and the business opportunities they afford. James is a seasoned technologist with more than 20 years of experience in distributed systems development and deployment, focusing on service-oriented architectures, cloud computing, and automation. Prior to joining SOASTA, he held leadership roles at Dell, Enstratius, Cisco, Cassatt, Sun and Forte Software. @jamesurquhart