Data Science: Next-Gen Performance Analytics


SOASTA Executive Chairman and Co-Founder, Ken Gardner, looks at the latest innovations in performance analytics and how data science can be used in surprising ways to visualize and prioritize improvements.


People that know me, my task here this morning is to do a ten minute talk on next generation performance analytics. If you know me, you’re quite skeptical about my ability to pull that off. First, some context – let’s talk about some numbers. The e-commerce market, specifically retail e-commerce sales world wide last year were 1.6 trillion, a T, not a B, trillion dollars. It’s expected to grow, essentially double, to 3.6 trillion in 2019. Every part of this market place is seeing very, very high growth rates. In the U.S. digital ad market place, we’re at about 70 billion, and we’re headed to 100 billion by 2021. There’s a lot at stake here. For a lot of companies, this is existential. You may or may not be here five years from now depending on your ability to make the digital transition.


Let’s talk about the elephant in the room: Amazon does this really well. They continuously monitor, measure, collect, keep, and exploit information about every user experience. They use that data to continuously improve what they’re doing. They improve the experience, they improve the performance. What’s important about them is that they have taken a very different approach to analytics than anybody before them. What they’re doing and how they’re doing it is very, very much worth studying. Traditional analytics dashboard systems, all of the ones that many of you are using. I was on Ghostery (by the way, great company) the other day and looking at a site, and it had 32 tags on that site. Those people are collecting a tremendous amount of data.


A lot of these products that were built in the early 2000s, were built before the cloud, and they were designed to sample and aggregate. The primary design rationale behind that was to keep the cost of compute and storage for that system inside of somebody’s budget. Now, because of Amazon and the other cloud companies, including Google and Microsoft and IBM, we’ve had just amazing leap forward both in terms of capabilities and in terms of the economics of being able to do this. At SOASTA we have collected, at this point, over 425 billion user experience beacons. We have every one of them. It costs me about $9,000 to store that – we’re talking pennies.


Werner Vogels, who is the CTO at Amazon, has been giving a talk since 2011 called Data Without Limits, in which he describes (using other companies) Amazon’s approach to analytics. First, collect and keep as much event data as possible – the red is my edits. If we take a look at a web property that gets 10 million pageviews a day, that’s going to be roughly 3.65 billion pageviews in a year, which is about half a trillion page resources that were components of those page views. That is a tremendous amount of data to store.


The last thing about collecting and keeping all of the data, is to deal with uncertainty. You don’t know what questions you’re going to ask, and you don’t know what algorithms are going to be invented, or that you might adopt, or you might implement, that will end up being very valuable to your company. In addition, you can change your mind and go back and redo it. Here’s our strategy, and has been from the very, very beginning: collect all of the data, all of the time, keep it forever. We collect a lot of data about page views. A previous speaker talked about the fact how much data you collect isn’t that important, and I agree with that. What’s important is that you take that data and you put it into a context that everyone on your team can understand.


Here’s an analytic dashboard. Let me talk about what’s in the upper right – it’s the conversation impact score. We take all of your page views, we reconstitute them as sessions, and we then determine which sessions converted, and we rank the page groups in your traffic in order of their importance to a conversion. Everything is not important, and everything doesn’t need to be optimized. This is a performance-free map. We’re going to drill down to the bottom. This is the product page – performance for that product page by browser, by browser version. Then we are going to go back up to the product page. One of the things that you’ll notice is the four most important pages in this website are: sku page, search page, browse page, and home page, and they haven’t been optimized.


In most engineering teams, if you give the engineering team a goal that says, “Please optimize our site. Please cut our page load times in half.” They will do it, but they will do it the easiest way possible, and they will do it in a way that might not be the most impactful. One of the primary use cases for machine learning is going to be to create context. You use machine learning to run all of your traffic to establish what’s normal. Then you want to be able to plot in real time normal juxtaposed to those tolerance bends. You fire alerts only when what happens with that metric, with that activity, or that revenue achievement, goes out of bounds. Essentially, we do this by the minute, you end up across a 24 hour day with 1,440 threshold pairs that will determine what the limits of your alerting are.


This is a single campaign launch dashboard, as I talk to people, as I travel around and I meet with prospects and customers, one of the biggest problems people have is they have a very large percentage of their marketing campaign fail at launch, and they don’t see it. The marketing analytics systems do not give you the information to determine that you just launched a $5 million email campaign, and that it’s failing. You get to find out tomorrow. That sucks, frankly. The idea here is to show you (in this case we zoomed in to a single campaign landing page) what happens to all of the traffic that goes off of that campaign.


The idea behind all of this is to give you context so that you can make good decisions and you can make good decisions quickly. I want to give a shout out to a guy named Cliff Crocker, who’s the person who introduced me to this. Cliff developed this at Walmart when he was there. Crocker charts, this is by device type, this is the desktop Crocker chart.


Session performance, conversion, and conversion rate from zero to 16 seconds. One of the things that you’ll notice here, as you look at this, the predominant percentage of your conversions occur inside of the median. Half of your traffic has a incredibly low conversion rate, and half of your traffic at two second, these guys have a 14% conversion rate, which is excellent. If you don’t believe that performance affects conversion, I hope this will change your mind. Same thing by browsers. – this is referred to as a Tufte small multiple. It is an excellent way, using the same scale, to show you differences across the dimension values in an important dimension.


Third party resource analytics. For most of you, more than half of the resources on your site are coming from third parties. I would tell you straight up, if you aren’t managing third party performance, you’re not managing performance at all. Thank you.