What percentage of the total page resources that your web or mobile property delivered yesterday were third party resources, such as ads, social media badges, monitoring scripts, or tracking tools? Exactly how big is the impact of third parties on key metrics such as conversion rates, bounce rates, and page views?
It’s surprising just how few digital businesses can answer these questions with any accuracy. Sure, everybody knows third party resources are a huge performance management problem, but the exact scope of the problem is unclear. So digital operations and development teams resort to resource budgets, extensive A/B testing of new resources, and a purely reactive approach towards managing issues with third parties when they arise. And they do. Often.
Enter SOASTA’s new Third Party Resource Analytics, a set of notebooks and reports that will soon come with Data Science Workbench (DSWB) and run against mPulse resource timing data. Built specifically to tackle the problem of understanding the scope of a digital business’s use of third party components, as well as identifying problem resources and resource providers, Third Party Analytics gives you insight into this critical front end performance component.
In this post, I’m going to walk you through some of what this solution package delivers to our customers, but first it’s interesting to explore the impact of third parties on digital businesses.
Third party resources are a growing source of performance issues
A recent article cited a study of the top 25 online media properties and the advertising resources on their page. The study pointed out that ads took up an average of 9% of a page graphically, but represented 55% of the bandwidth used by the page, and 54% of the total load time. The problem was so significant, it led New York Magazine to issue an ultimatum to advertisers:
“If the advertiser provides creative that does not load in a certain time, then there cannot be viewability conversation,” said Ron Stokes, executive director of client advertising solutions.
Viewability is increasingly the metric that advertisers are demanding be used to confirm an ad is delivered, therefore triggering a charge from the web site. So if your resources consume too much time and bandwidth on New York Magazine’s web site, you won’t get paid.
But ads aren’t the only third party resources with a significant impact on user experience. Many online retail customers see significant performance impact from their customer experience monitoring tools, such as Adobe Analytics (formerly Omniture) or IBM Web Analytics (formerly Coremetrics). Some are seeing problems with oversized images being delivered from third party content providers. Not necessarily all the time, but when it does happen, it can have a significant impact on customer experience.
But getting an accurate picture of how these resources impact performance is problematic. It requires collecting data about every resource request and related timing metrics–around the world and on all device platforms. It requires analytics tools that can ingest this vast amount of data, and analyze the data to give insight into where problem areas lie. This is what SOASTA Third Party Resource Analytics aims to achieve.
(While this solution package is focused on third party resource analysis, it should be noted that the library it leverages can do similar analysis against first party data or combined first and third party data.)
What is the scope of your third party resource challenge?
The package consists a set of DSWB notebooks, including a summary notebook that gives a good overview of what can be done with the analytics, and supporting notebooks that allow users to drill into the data from a variety of different dimensions, including device type, resource type, and page groups. The package also includes the new RT library, which contains a variety of powerful functions and visualizations. The RT library can also be build new notebooks and reports, if desired.
The first function you’ll see in the summary notebook is perhaps the most powerful of all: a table listing first and third party volume and percentages by page group and for the property as a whole.
There are other ways you can break down the same data – by device type, for instance – but the default breakdown by page group is extremely useful for understanding how third parties impact not only your site as a whole, but each key function within.
Understanding the scope of third party resources on your site is critical to getting a handle on where to find problem areas. Are some page groups more third party dependent than others? Is the web site as a whole dependent on third party performance to meet or exceed user expectations? The table above goes a long way to answering those questions.
Where are your third party problem areas?
Now that we understand how our digital property depends on third party content, we can determine where third parties are impacting user experience in negative ways. The summary notebook starts this process by looking at third party resource timings by device type in two ways.
The first is by a tree map that allows the user to click on a device type to see all page groups with third party components delivered to that device type. Another click on the page group shows all resource providers for that page group on that device type. The size of the boxes is the volume of resource requests, and the color indicates the load time for those resources.
The other way of understanding resource timings for a variety of dimensions, including device type, is via a box chart. This box chart shows resource load times by resource provider domain. The bars at the top and bottom indicate the minimum and maximum load times observed. The box in the middle is the middle half (i.e. the second and third quartile) of load times. The line in the middle of the box is the median load time.
Box charts are a powerful way to see whether a provider is largely consistent with a few outliers, or has highly inconsistent load times. The bigger the box relative to the bars, the more spread out the observed resource load times.
Note in the chart above that most resources have boxes close to the bottom of their bars, but there are a couple with significant outliers as well as relatively large boxes. These are the providers to focus on first.
These features of SOASTA Third Party Resource Analytics enable your business to quickly identify where potential third party problems exist, but how do you identify specific actionable solutions, either with resource providers or even individual resources?
How can you efficiently troubleshoot third party issues?
Breaking down problem areas into specific actionable insight comes down to two key elements. First, which individual resource calls are having the most impact on page performance? Second, how are resource providers behaving overall, and are there specific issues with the way their content is delivered or received. Usually, these questions should be asked in the context of some other filter, such as a specific device type or browser type, or even page group.
For example, by examining the worst performing third party image resources in a page group, along with how often each resource is requested, developers can focus their energy on the images that are having the most impact on user experience. In the graph below, the blue bars are median load time, and the green dots are request volume for that resource.
While the first five items are the slowest, the next two items have a bigger impact in terms of the number of users affected, so it’s probably a good idea to focus on those items first.
For analyzing how resource providers perform overall, SOASTA Third Party Resource Analytics provides a histogram view of volume for each few milliseconds of load time.
This is hugely useful in identifying where things like caching and network latency are affecting load times, as well as identifying services that struggle to deliver content consistently (usually denoted by extremely long tails, as in the graph above).
Next steps: Getting started with Third Party Resource Analytics
SOASTA Third Party Resource Analytics will be available with every Data Science Workbench installation or upgrade at the end of this month.
About the Author
James Urquhart is Senior Vice President, Performance Analytics at SOASTA, Inc. Named one of the ten most influential people in cloud computing by both the MIT Technology Review and the Huffington Post, and a former contributing author to GigaOm and CNET, James brings a deep understanding of disruptive technologies and the business opportunities they afford. James is a seasoned technologist with more than 20 years of experience in distributed systems development and deployment, focusing on service-oriented architectures, cloud computing, and automation. Prior to joining SOASTA, he held leadership roles at Dell, Enstratius, Cisco, Cassatt, Sun and Forte Software. @jamesurquhart