Velocity NY 2016 Tutorial – Measuring Real User Performance in the Browser

To people responsible for the health of web properties, monitoring, measurement, and analysis of real user experiences is key to delivering a consistent and delightful experience. Drawing on their experience with the open source boomerang project, Philip Tellis and Nic Jansma explore various tools and methods available to measure real user performance in the browser.

•  APIs: Navigation Timing, Resource Timing (Levels 1 and Level 2), User Timing, Error Logging, and more

•  Measuring full-page apps

•  Measuring single-page apps

•  RUM SpeedIndex

•  Measuring continuity

•  Hacks when an API isnt available

•  Avoiding the observer effect

•  Filtering data to get rid of noise

•  Identifying outliers and treating them differently  browser

Speakers: Philip Tellis, SOASTA | Nic Jansma, SOASTA




Philip: We have about a minute, so we’re just going to introduce ourselves now so we can get started with the tutorial. My name’s Philip, and this is Nic. We’re both from SOASTA, and we work in an open-source JavaScript library called boomerang. There’s a link to boomerang right here. What boomerang does is measure user performance from the browser as users browse through our site. What we’re going to talk about today is all of the things we’ve learned over the last 6, 7 years building boomerang. Essentially, how do you measure everything related to performance and a user’s experience with that performance within the browser?


Nic: I’m Nic, nice to meet you guys. We do have slides online right now. The link’s on the very bottom right there. We both Tweeted up the link as well. Feel free to follow along at home.


Philip: Before we get started, what’s the experience here in the room? Who considers themselves a beginner in performance? Intermediate? Experts? What about with JavaScript? Beginners in JavaScript? Intermediate? Experts in JavaScript? We’re going to do a lot of JavaScript here and a lot of performance. There’s not going to be any back end level stuff because it’s all about browsers. Do you want to go over the agenda and then I’ll start?


Nic: Sure, we have 3 hours today. We’re going to cover a lot of topics. Feel free to ask questions if you have at any point throughout. We’re going to start out just what is user monitoring? We’re going to cover the browser and performance CPIs. We’ve got a lot of coverage there. We’re going to talk about the visual experience, so what your users are seeing when they’re loading your site. We’re going to talk about how to beacon this data, so when you get the data, what to do with it. Then we’re going to talk about single page apps and what to measure after the page load, along with noise in the data. We’re probably going to take a break in the middle, I think, at 10:30. That’s the agenda for today.


Philip: Our story starts in 1747 with this French clergyman named Jean-Antoine Nollet. He used to like playing around with physics, and at the time electricity was pretty hot. He actually invented one of the first electroscopes. We still use them today, just call them beacon collectors, and they run on Amazon. They’re about the same thing. He ran this experiment in 1746 where he wanted to measure how fast electricity traveled and how far it traveled. He shot an electric current through a chain of 200 monks connected by electric cables and measured how long it took them to jump, and thereby measured the first electric signal using real users. Our methods today are a little less intrusive. We don’t shoot electric currents through users, but we do send signals to their browsers and get them back in order to measure how long it takes to load their pages.


  Before we get into how do we measure, the question is why do we even care? We’ve gone through many years of web development without measuring how long it takes to load web pages. We’ve done a lot of server-side measurement, but how long does it take to measure users’ experience? We haven’t really cared about that. The reason we care is because there’s a difference between delighting the users and frustrating them. The experience you provide to them, whether it’s about speed or about the ability to complete a task, or the availability of certain products on your site, they have the ability to delight or frustrate the user. Depending on what you do, the user’s reaction whether they actually buy something, whether they come back to your site, whether they are repeat users, whether they tell their friends about your site, all depends on where they fall on the scale.


  We have various variables that feed into this responsiveness, smoothness, and cognitive dissonance, or resonance that feed into this whole scale of delight to frustration. It’s important to know where you actually fit on each of those inputs and each on this output scale to know whether you’re actually providing a useful experience to your users or not. Measuring that experience directly, where the user is from the user’s point of view, is very important, which is why we spend a lot of time trying to figure out how to measure these things.


  How do we actually go about measuring? Fortunately for us, we live in a time when the browser has a lot of APIs that make it easy for you to measure performance. We now have a high-resolution timer. We have navigation timing, we have performance timeline, resource timing, user timing, and many other APIs. We’re going to go into most of these in detail in further slides. Essentially it means the browser gives you a lot of information out of the box, so you don’t have to write a lot of hacks. There are still going to be a lot of things that you do need to build on your own, but we’re going to start off with just using what the browser provides.


  The DOMHighResTimeStamp, who’s familiar with the HighResTimeStamp? Not a lot of people. Are you familiar with the Date object in JavaScript? The Date object is what we’ve traditionally used, boomerang has used it for many years to actually get the time. The issue with the Date object is it’s not monotronic. If their system’s clock happens to go backwards, for whatever reason like they change timezone or the battery died and the system clock reset, the new Date object is going to also go backwards in time. High resolution time stamp is monotronically non-decreasing, which means it’ll never go backwards. It’ll only go forwards. Deltas always tend to be positive. It also gives you a better resolution. A new date gives you millisecond resolution. The HighResTimeStamp gives you sub-millisecond, typically microsecond, resolution represented as milliseconds, and the fractional part is microseconds.


  Some examples of what the 2 look like. Now the big difference is that new date is always relative to the Unix epoch, which is January 1st, 1970 UTC. The HighResTimeStamp is relative to the start of navigation, so performance that timing that navigation start. If you look at DOMMHighResTimeStamp at any point in a page’s life cycle, it’s going to tell you how long since the users navigated to that page that that time stamp is.


Nic: Which is also just very nice when looking at these time stamps because instead of having to do looking at a number in the 1.4 trillion for today’s milliseconds since 1970, which is in the thousands or hundreds. Just a lot easier number to visually look at.


Philip: Who can guess what that time stamp on the left is? Not me. What we meant by monotonically non-decreasing, if you look at the examples here, navigation start performance at now 0. We have a sudden time for Date.getTime. If you go down you see at 300 milliseconds when the users clock moves back 1 second, Date.getTime also moves back 1 second, while performance at now it does not, it actually moves forward because this is counting relative to navigation start and not relative to the system clock at all. That’s a very important distinction. When you’re actually measuring relative things in the browser, you want something that does not move backwards, that’s consistent, that’s moving forwards. How do you use it? Something you can do in trying your browser right now, open your WebDev console. Just type in and it should tell you exactly how much time has passed since you opened a web page to now.


  Next thing we’re going to look at is the navigation timing API. Who’s familiar with the navigation timing API? A few people. Navigation timing API has been around since maybe 2010. Internet Explorer 10 was the first … Was it 10 or 9?


Nic: 9.


Philip: 9 was the first browser to implement it because of this guy. What it does is tell you different time points within the page’s loading cycle. From the time that navigation start is when the user indicated they wanted to load a page either by typing in a URL in the browser or by clicking a link or following a bookmark or something else. From that point, there’s various things that could happen that could be redirects. There could be reading out from cache, there could be a DNS lookup, there could be a TCP setup time. The request response finally downloading the page, the different stages of downloading when the browser’s creating the DOMM until all the scripts, images, and everything else is loaded. That’s when the load event files.


  In fact, there’s also a difference between load event start and load event end. Load event start is when all the assets have completed. Load event end is when all load event handlers have finished executing. This actually gives us a lot of insight into the actual loading of the page. You can tell just by looking at these various time stamps whether you had a problem on the back end …


  By looking at these various time stamps, whether you had a problem on the backend or the frontend, right? Did I have a problem with DNS look-up? Or did I have a problem with making a TCP connection? Did I have a problem with too many re-directs? All of that information is very readily available within this browser’s API. Nic’s actually going run this in the browser right now. Let’s open the performance object, where we see the timing object in there. Under timing, we see a whole bunch of time stamps. Again, something you can try in your browser right now.


  These are absolute time stamps. Using the high-resolution timer, but relative to the Unix epoch. What you can do is do deltas in between these. If we do, for example, load event start minus navigation start, it tells us the entire time it took to load the page. If we do something like response start minus request start, it tells us how much time it took just to do the backend.


Nic: What was the …


Philip: Response start minus request start. Maybe go up a bit. I don’t know if everyone can see that, but right at the bottom. The response start minus the request start. 585 milliseconds just to make the backend request. The time from sending the request to the server to the time the first byte comes back to the client. Now, that doesn’t tell you things like how long did it take to actually do the sending? How long did it take to do the processing on the server? How long did it take to do the downloading? It just tells you that’s the bulk. Anything else you wanted to know, like how long did it take on the server? You need to send time stamps from the server.


  Another thing is the performance and navigation, which tells you how many redirects happen. In this case, there was one redirect. We made a request. The server did a redirect, like 303 and 304. 301 and 302.


Nic: 1 or 2?


Philip: All right. All of these, just looking at this, are indications whether you are doing something wrong on the server that you may not have known about because in developer testing you were using ideal conditions, whereas your user is not using ideal conditions. Things like redirects or DNS look-ups are happening on their side, but you don’t know this on your side. Looking at navigation type is another interesting thing. You can tell whether a user came straight to your site or did they click on the back button or did they do a reload, which are important things because they can tell you whether they have a full cache or an empty cache or whether unload handlers have already fired or not.


Nic: Right.


Philip: How do we use it? Again, you can call we noted performance step timing at any point within the page’s lifecycle. We prefer to call it in the unload handler, because that’s after most of those events have happened, and therefore those time stamps will actually be set. If you call it before the unload handler, a lot of the intermediate time stamps will be 0, because you haven’t reached that point yet.


  Browser support is pretty good. At this point in time, almost every browser, we have 90% global and 95% in the U.S. support the navigation timing API. Internet Explorer doesn’t show that, but from 9 onwards it does support it. All the other browsers … Well, Safari is actually very recent. Right? Safari started in …


Nic: 2009.


Philip: … 9.1 or 9. That’s been the latest browser. Safari and IOS Safari to add support for navigation timing.


  How do you integrate this into your site? Well, you can send the data back to your server for logging. Then, do interesting things. Or you can use a bookmark written by Andy Davies, sitting right here in the front row, to just show a waterfall within your page itself. You can use Boomerang, which is what we work on to collect the data and send it back to your server. There’s a couple service side implementations, like Boomcatch and Boomerang Express that collect the data from Boomerang and do interesting analytics on it. Extending it to Graphite or StatsD. There’s a tool called, which is a mash-up between web page test and a whole bunch of other tools to do synthetic testing. It uses navigation timing. There’s another thing called Piwik, which is an open-source clone of Google Analytics, that can zoom navigation timing data. There are people, although Google Analytics also sends it, a small sample of navigation timing data. There’s also commercial companies that use it.


  Our company, SOASTA. We built a tool called mPulse that can consumes navigation timing data and does interesting analytics. Google Analytics, SiteSpeed, New Relic Browser, NeuStar, Speedcurve, et cetera. A whole bunch of companies that consume navigation timing. It’s considered the defector standard at this point. It’s been around now 6 years. We assume that it’s always available.


  Some tips when you are using navigation timing. These are things we learned the hard way. Use fetch start instead of navigation start, unless you care about things redirects or initialization of cache or cabs, et cetera. Navigation start is everything. Navigation start will tell you what the user experience. fetch start will tell you what you can actually control. fetch start to response start, for example, tells you things like DNS look-up, TCP, connect time, serve time, et cetera.


Nic: One important scenario there too, like why you would use fetch start instead of navigation start. If your users, for example, are coming from a search result page to your site … Let’s say you’re on a Google search result and they are clicking on the first link, Google will often do a redirect back to themselves to track that you clicked on that link and then send the visitor to your start. If you’re starting at navigation start, you’re actually timing how long it took to redirect to themselves before they went to you. You’re not just measuring how long it took to load your site. Excuse me.


Philip: There’s also the case where some sites have an unload handler that does a lot of work. In that case, you would actually see the time it took to run the unload handler as well.


Nic: Right.


Philip: Load event, and in fact any of the time stamps, like I mentioned, time stamps that have not yet happened when you take a reading will be 0 at that point in time. If you delay a reading further, you will get those time stamps. We don’t have an accurate way to measure request time. That’s the amount of time that it took for the request to get to the server. We have often thought we could send the server time stamp when it actually received the request, but the problem is they are running 2 different clocks so the server clock and the client clock cannot be compared. You can’t just do a delta everything and say, “This is what we think the request time was.”


  There are things like time stamps up to a response end maybe 0, because some browsers have preemptively have pre-fetched content. We don’t know yet how things work with pre-render. Do we have any data on that? We’ve seen various results. Is everyone familiar with pre-render? All right. Pre-render is a method for you to tell a browser, particularly Chrome and I think Chrome and Internet Explorer at this point, have the ability to download a page before a user actually navigates to it. Download all the images, scripts, CSS, surrender the page and keep it available in a hidden tab, so that when the user actually click the link, it just makes it visible. We don’t yet know how navigation timing works in that scenario. We know that the future versions of this spec are going to include pre-render time points within spec.


  If possible, do any beaconing as soon as possible. Too many uses of possible in there. What happens is a user could leave your site before it is completed loading, right? They might load up the page. Either it’s taking too long to load or the thing they wanted to see … Sometimes you search for a definition of some term and the dictionary page comes up and before the whole page has loaded, you’ve already seen what you need to see, so you just close the tab. If you want until unload to actually send your beacon, you’re going to lose that beacon. It’s good to send the beacon as soon as possible and then maybe send another beacon with complete results later on.


  This is not suitable for single-page apps. That’s one important caveat to note. If you have a single-page app that has one full navigation and then lots of other end page navigations, navigation time is only going to show you that first navigation and nothing else. It stops at load event. Anything that happens after the unload event is invisible to navigation time.


Nic: We’ll talk about single-page apps later in.


Philip: Things that are coming soon to navigation timing. It’s going to be part of the performance timeline. The performance timeline was a spec was in development just after navigation timing came out. The things we learned from navigation timing and things like resource timing and user timing are part of the performance timeline. Navigation timing has not yet made it into that. That’s coming soon. We’ll be able to do something like performance to get entry’s navigation, which give us common interface through all performance timeline.


Philip: … which gives us a common interface to all performance timeline things. Support for HighResTimeStamp, which means we’ll also see microsecond level resolution. Like I said, pretender, NextHop Protocol, which is something we’re very excited about because it tells us whether the servicer supported HTP2 or SPDY or what kind of protocol. Right now there’s no way for us to tell from JavaScript unless you’re using Chrome, whether the server was a H2 or HTP1. Then things like transfer size, encoded body size, and decoded body size which are important to tell if your server-sent content is up to naught, whether the headers were too big, whether the content was optimized for the user’s device or not. We are very excited about these things and just waiting for them to be available.


  Something else that has been available just a little bit longer than navigation timing. Chrome before navigation timing, the spec was complete. Chrome added something called Low Times. You can actually do this right now if you have Chrome, go and type in Chrome Chrome.loadTimes(). It should give you an object that looks like this, which has a lot of the same things as navigation timing, but one of the things that we were very excited about was that firstPaintTime in there. It tells you, and we’ve checked with the Chrome team. It turns out it is first meaningful paint. We’ll actually cover these different types of paints later on. First meaningful paint is actually what you care about a lot. It’s when do the users have something meaningful on screen that they knew that the page was loading.


  Then also some interesting things, like I said you can tell on Chrome whether they came in from HTTP2 or not. That’s because you have this connection Info field in there. In this case, we see quic/1+spdy/3. That’s basically telling us that instead of HTTP, it uses the SPDY protocol which Chrome supports. Instead of TCP, it uses the QUIC protocol, which again, Google developed as a faster alternative to TCPIP. It uses UDP instead of TCP to send back its class and eliminate the need for acknowledgments.


  These are interesting things because it can tell you. You can look at your data and see did users on QUIC have a faster experience than users on TCP, or did users on SPDY or H2 have a faster experience than users on HTP1. Accordingly, depending on how many users use that, you can optimize your pages accordingly.


Nic: Importantly too for this list, these attributes aren’t very well documented anywhere. I wouldn’t necessarily rely on them too much. They can give you some hints towards different times, but as far as I can tell they’re not in any official Chrome documentation anywhere so they can change at any time.


Philip: Yeah. When we first got these information’s we had to look through the Chrome source code to … not the Chrome, what was the Chromium browser source code at that point.


  A good question is, what did we do before navigation timing. Until very recently Safari didn’t support it and still a lot of users are IE6, 7, 8, mainly corporate users that are tied to a corporate backend system that requires IE7 or IE6. What did we do before that?


  There were hacks we had to do. We would hook into the before Unload, the unload, and the page hide events. Whenever any of those events fired, we would store a cookie that included the timestamp using newDate.getTime() and the URL of the page. Then in the onload or page show event of a subsequent page we would read the cookie, and then we would compare the URL with the document.referrer. If the two matched, then we would take the two timestamps, and take a delta, and say that was the time between the user requesting the page and the page being available.


  Now, the reason for comparing URLs is because the user could go off site and then come back, but that cookie would still be available. That made sure that they actually came from one page on your site to the next page. Something to note, this obviously doesn’t work if the user came in from an external site like they came in from a Google search results page because we can’t set a cookie on Google. You can’t tell very detail information but it turns out we could make a lot of guesses. The before Unload event corresponds to the navigation Start timestamp. They typically fire at the same time.


  The unload and page hide event, so for those of you who don’t know the page hide, it’s exactly the same as unload, except some browsers support page hide, some browsers support unload. When a browser writes a JavaScript state to cache and reads back JavaScript state on a back-navigation, it would use page hide and page show. If it does not write that state to cache it will use onload and unload. That’s literally the only distinction. You don’t need to worry about anymore. Just think, if I’m going to attach a handler to onload, I also need to attach to page show. If I’m going to attach a handler to unload, I also need a handler for page hide.


  Unload and page hide correspond to response Start. It turns out the browser does not fire the unload event until it’s received the first bite of the next page. That’s unless you’re actually quitting the browser and there is no next page in which case it may or may not fire the unload event. Just keep your fingers crossed.


  As you can tell, these things were not very reliable, but they worked in about 85% of cases. We also had to hook into the click event because it turns out people can also navigate by opening a link in a new tab or a new window, in which case there is not unload event because you’re not unloading the current page. We hook into a click and a form submit, look at the URL of the form or the link you’re clicking on. If the next page has the URL that was clicked on, then we say, “All right, the user came from that link-click and therefore this is the amount of time.”


  This doesn’t work for external referrals, but surprisingly it worked all the way back to IE5.5 running on Macs on which nothing worked. We tested that. All right, I’m going to turn it over to Nic to talk about results timing while I get some water.


Nic: First of all, were there any question about nav-timing at all? Anything you want to discuss there before we go on? You can always ask later too if you want.


  Okay, so the next browser API that we have is resource timing. It’s very similar to navigation timing. We get a lot of the same timestamps as you would with navigation timing for resource timing. Resource timing instead of the navigation, we’re actually measuring all the different resources on the page. We’ll come back to this time series in a bit, but this is the inspiration. This is like your Chrome DevTools, whatever dev-tools you’re using.


  The waterfall. In the waterfall, you get to see all the different resources that were loaded on a page, how long they took, the breakdown of how long it took to do the domain lookup, or to connect to the server, or to download. You get this series of resources that are being fetched in chronological order going down. You can see how long it’s taking to load everything. Every one of these resources contributes towards the total page-load time.


  That’s the inspiration for resource timing is how can we get this data which you can do on your local machine when you’re developing from all the visitors in the wild as well. For every single visitor that’s coming to your site, can you understand how long it’s taking to load the different resources, all your CSS, all your images, all your JavaScript?


  Resource timing gives you a list of all the resources that were fetched, and it gives you all the timestamps that you would expect. The start time is the very first moment the browser tried to fetch this resource from the network. It may redirect through a couple different URLs to get there. It may look into the cache to see if it’s already in the cache. It may take a couple milliseconds to actually load it from the cache. If not, then it has to do a domain lookup, a connection to the server has to request the resource, and has to get the data back.


  These are all the timestamps that you would get for every single resource on your page from the visitors perspective. For things like from response Start to response End, that’s the total download time of the resource, how long it took to fetch it from your server. From request Start to response Start, that’s how long your server took to process, to think about what resource to give you, fetch it from its disk or whatever, and send it back to you.


  Resource timing gives you all this data from JavaScript. You can capture, you can compress it, and send it back to your servers for big data analysis, that kind of stuff. Let me show you an example. Okay, so back to the DevTools console. Phil talked about the performance timeline a little bit, but we’ll get into that a little bit more. The performance timeline is a way of capturing different performance metrics that happen on the page, chronologically


  on your page, and different browser APIs can add data to this performance timeline.


  In this case, you access the performance timeline via getEntries() or getEntries() by type or by name. It’s just a list. It’s a list of events that happened on your page. In this case, it’s a list of resources that were fetched, so this is the resource timing data. He opened up the very first resource on the page that was added to the timeline. You can see all the data that I was talking about. The start time, it took this CSS, for example, started at about 10:20, so one second …


Philip: For example, started at about 10:20, so one second or so, within the page load. The duration was 68.766, so it took about 68 milliseconds to fetch the CSS from the server. You also get a breakdown of other things, response, or all of the different time stamps within that resource load, you get it from resource timing. It is a fantastic way of being able to rebuild the waterfall from JavaScript, you get all of the data from JavaScript. We will go back into some of the data that we got in a moment.


  How could you do this before resource timing. You kind of could, you kind of couldn’t. You could measure from JavaScript, how long it takes some resources to fetch. For example, if you wanted to measure how long it took for an image to be fetched from a remote server, via JavaScript, you could create a new image … image equals new image. You could change the source of that image to the image that you want to fetch, add a load handler to it so you know when it is fetched from the server, and then you can compare those time stamps.


  This is a way of actually measuring, without resource timing, how long it took to fetch something from a remote server.


Nic: We have a question.


Philip: Yes.


Philip: I will tell you why 150 in a little bit as well.


Nic: Just to repeat the question, we showed in the def tools, was it a list of network requests? And the answer was yes. We are going to talk more about it in a little while. One thing to note about this JavaScript code is even though we attached the load handler and it looks all right, the problem is that the load handler could get delayed if the browser was busy executing other JavaScript. It is not guaranteed to actually fire when the thing completes loading. It also will not tell you when, it tells you when it was loaded but not when it was actually rendered on screen, which is a whole other problem that we might talk about a little at some point.


Philip: On top of that, you only get the total duration. You cannot measure DNS timing or how long it took to connect to the server. And it is not practical. You could but you wouldn’t want to construct your entire web page via JavaScript, that is what HTML is for. If you were to do this, if you wanted to measure all of the content on your page just from JavaScript, you would actually have to incrementally add all of these things to your website via JavaScript, instead of just putting them in the HTML.


  That is how it was done before. Obviously it is not practical for doing it. That is why we have resource timing today, is performant way of capturing all of this data automatically, you don’t need to do any manual instrumentation. You just get it for free. The performance time line, which we showed you a demo of, as I said, it is a standardized interface for collecting all of these different performance metrics. One of the things in there is resource timing. There is also user timing and some other thing.


  One of the main things in the performance time line today is all of these resources you can get. There are three main ways of accessing this data. There is Get Entries. Get entries by type and get entries by name. Get entries just gets all of the entries in the timeline, resource timing data, user timing data, whatever is in there. Get entries by type is where you can filter down by a specific type. If you just wanted all of the resource timing data, you can do get entries by type with resource. And if you wanted to look for a very specific thing within the time line, you can do get entries by name. In this case you can actually use the URL of the resource you’re interested in.


  If you have a specific CSS or JavaScript or something that you care about, you can do get entries by name with that URL, and as long as it was downloaded, you would be able to see that. Let’s jump back to the demo.


Nic: We also have some code in Boomerang that will let you get an entry by wild card. Which you cannot do with some of the built in API’s because it is a little slower than the built in stuff but it is cool when you just know I want things that are from this domain, get me everything from this domain.


Philip: We showed you this but this is getting all of the entries within the time line. You can see there was a 150 of them in this case. You can iterate through them in the Chrome Dub tools. If you just wanted to look for resource timing data, which will give us the same data in this case, you would do get entries by type and then you pass it a stream of the data that you want to get. In this case, the known stream that you have to use is resource and you get an array. A list of 150 things. You can do all of those standard array options to get stuff out of it. This is the data that we showed you before.


  The other thing that you can do, and I will go into what all of these metrics mean in a moment, but the other thing you can do is you can actually just get very specific entries out of it. If you are just interested in one URL, we will use get entries by name, and then pasting the URL that we had before. This is the URL on the link that we saw before, and it will return an array of matching things. It is not just a single thing because you can actually download the same URL more than once.


  In this case, we did find a match. This is that one URL that we fetched. We knew that the Url had a time. Maybe it was your header image or your main CSS for your website. You can get the data for just that one URL instead of having to iterate over all the 150 things there. You can see that we get a lot of nice time stamps. In this case, different from navigation timing, that are all based on the difference from the start of the navigations. They are all of the high res time stamps. Instead of a number in the 1.4 trillion for the time stamp, you actually get 1,000 which means a second.


  This is what we saw in the console. There is a couple of different parts to it. There is time stamps and there is some characteristics of the navigation. This is one element within the array. The name is the URL, the resource, it is not called the URL it is called the name, just for compatibility reasons with the performance time line. You get the initiator type. The initiator type tells you why the resource was downloaded. It doesn’t tell you exactly the element that started fetching it but it tells you the type of element that started fetching it.


  In the last case you could see that it was IMG. An image element wanted to download through PNG. These are all of the different types of initiators that you would see in the wild. You have IMG for an image in the page. Link would typically be your CSS. Script is obviously JavaScript. If your CSS files themselves, have a URL or an at import, than you will see CSS as the type. Important distinction there. The link type is downloading the CSS within the page. The CSS type is anything within that CSS sheet, downloading additional things. You can see fonts and URL in the import as the initiator there.


  XMLH could be a request for XHR. If you are using SPG’s you will see the full IMAGE as the initiator for that. If you are still using flash for some reason you might see object as a type. It is basically the name of the element that started fetching this resource. Why was this kicked off. What element kicked it off.


Nic: There is also some interesting things like if you have a link rel equals pre render on DNS pre fetch, we don’t know how that it is going to work out in the sense that it is going to show link or is it going to show pre fetch or pre render in your initiator type. Those are things that are still being worked out.


Philip: The question is what if the image is a background image from a CSS. In that case it would probably be CSS because you designate it with your URL with the URL for the background image. It is kind of confusing. The CSS itself kicked off the resource so it would come in as CSS as opposed to image. It is not actually, and that is the very important distinction, it is not the content that was fetched, like it is not the content type, you don’t see PNG on there. You are seeing the type of the HTML element or the XHR or the CSS that kicked off the image instead.


Nic: It is important to note that in that case, if you want to know if CSS background image, you cannot actually tell just by looking at this thing that whether it was the background image or another CSS file loaded while import. Because they both have the exact same initiator type.


  Then the rest of the things in here are basically time stamps. They fit into the time line that we …


Philip: They fit into the timeline that we initially looked at. Basically every single one of these phases within the downloading of the resource is listed within that object. Along with the duration which is the total duration for the whole thing.


  Okay, there’s an important thing to note about the resource timing buffer and this goes back to your question about 150 elements when we saw it. Now by default, to conserve memory, only the first 150 resources are kept within the resource timing buffer. This is so if you’re on a SPA website, if you’re on a dynamic website that’s consistently kicking new XHR’s to grab new data. If we didn’t have some sort of limit, the browsers memory would grow unbound even if nobody was looking at this data. We don’t want that so, by default today, most browsers in practice have a limit, if you don’t change it, of a 150 resources. So the first 150 resources on the page go in here. After you’ve hit that limit, nothing else gets added.


Nic: Yes, it’s like one of those things where a new API comes out and says “Oh wow. It’s this new cool API I can use.” and then you go “Oh dammit, it’s only works halfway there.”


  I have 200 resources on my page.


Philip: Yeah, I mean the average website today loads 100 resources. Many of them load 200, 300, even more than that depending if they have ads and stuff like that. Depending on your usage patterns, you may want to increase this. You can increase this from the default of 150 to whatever you want. There’s a set resource timing buffer size and you just give it a size of 1000 or something like that. You can also manage the buffer. You can clear it, if you want. Like if you want to start clean at a certain point within your app, you can clear the buffer. You can listen for when it gets full as well if you wanted to take different action then. It’s important to note, don’t just increase this to like a billion because if your app is up for a long time on someone else’s computer, and you’re just constantly collecting these resource timing entries, you don’t want to just grow the memory unbound. If you’re not going to do anything with it, don’t change the buffer size. If you’re going to be looking at the data, capture the data. You can clear it, if you want.


  We’re going to talk about beaconing this data later like how to get it back to your backend but if you have this resource timing data, what are you going to do with it? One thing that you can do, if you want to collect all the data, and want to rebuild this waterfall on your local machine, you could, for example, just convert it to json, collect that data and just shoot it back to yourself. Let’s take a look to see what happens if I were to just convert all the resource timing data on the page to json. So we can just do a json.stringify. Okay. Scroll, scroll, scroll, scroll, scroll. This is, I think, the Google slides website is what we’re on right now. That’s a lot of text. There’s a lot of data there, right? There’s a lot of different time stamps. There’s a lot of different URLs within that. In fact, if we look at the total size, it’s 83 kilobytes worth of resource timing data. It’s probably just not really practical from a beaconing point of view of actually getting all the data back to your background. It’s a pretty big payload.


  The average website, as I mentioned, when we looked at HP archive a little while ago, was around 103 resources per page and the average URL link is around 100 bytes … 85 bytes to 100 bytes. On average, you can expect at least about 45 kilobytes worth of data, of resource timing data for every page load. If you’re going to be sending this data back to yourself, you want to be conscientious of both the users bandwidth, how much time it takes to stringify all this data. You want to send it out as quickly as possible because the longer you delay or the longer that it takes, the less likely the data is going to reach there because the user could be closing their browser or networking things could get in the way.


  The default TCP window, so the default amount of data that you can send before you need to get a response back from the server is about 15 kilobytes. It’s a good … in general. It’s a good practice if you’re going to be sending data back to your backend for your analytics later to try to keep the data as small as possible, underneath around 15 kilobytes. That means you can just send it all in one verse. You don’t even really have to wait … the browser doesn’t have to wait for it. If it’s more than that, then your browser has to wait for acknowledging from the server before it can send the rest, everything beyond 15 kilobytes.


  What can you do? We have a way of … we have an open source script that you’re welcome to use. It’s called resource timing compression. We do a couple tricks with the data to really compress it down to a much smaller size. Something that’s much more manageable to send back to your backend. An example, resource timing data object on the left and let’s say you had three of those, three resources that you had on your page. On the right, is kind of the data structure that we’ve compacted down to. We compress the URLs by using a smart tree. We compress a lot of the time stamps by reducing the precision, like we’re not tracking microseconds.


  We change numbers to base 36 just to save some string size, etc. we have a bunch of different tricks that we’re playing. The good thing is we can get this resource timing data to about 15% of the original size. So we’re compressing it down 85%. It’s a much more reasonable pay load to send around, to send to your backend. The open source script will obviously let you uncompress it as well. If you’re interested in playing with this data, take a look at it. It should save the amount of data that you’re actually be sending around.


Philip: There’s other ways of compressing your data, right? Feel free to do it in whatever way makes sense for your data, for your website. I mean if most of your stuff is all on the same domain, you know which domain you’re talking about, you can have a map of domains to like a number or something like that too. There’s a lot of different ways of compressing the data.


  One very important thing to know about resource timing if you’re looking at it in seriousness is the timing allow origin HTTP header. By default, any resource that is loaded from a cross domain … a cross domain is any domain that is not the domain that the page is loaded from. If your domain is and you happen to be loading your images from, that’s actually a cross domain. Even though you own it, even though it’s like a sub-domain of your domain. For browser security purposes, it’s considered a cross origin domain.


  By default, resource timing will hide all of the details, all of the sub-phases of resource timing for cross origin domains. You would still get entries within the resource timing data array for resources that are loaded from other domains but they’ll only have the start attribute and the response end attribute. So basically the beginning and the end. You wouldn’t actually get a breakdown of the TCP timings, the DNS timings, when the request was started and all that kind of stuff. You get the total duration but none of the breakdown of anything in between. This is for privacy concerns. There’s some interesting attacks if you know some more details about how long it took to look up the domain and stuff like that. The good thing is that you can actually effect this behavior, if you want. If you own the server in question, if you own the server that’s sending the resources so the cdn, for example, If you send this header, timing allow origin, which in practice, you’re probably just set to star as the value. Then that behavior is overwritten and you’ll get the full breakdown of the timings.


  That requires you to modify your web server or whatever sending this whether it’s a apache, nginx, or what have you … or CBN, you can probably often set an additional HTTP response header on your content. The reason it’s important is that the duration attribute includes blocking time.


Philip: Duration attribute includes blocking time. Actually, I’ll get back to this in one moment. A lot of third parties today, so like your Google analytic script, Facebook, etc., they’re adding this header to the content as well which is great. That means you can time how long it’s taking for all of these third party scripts to be added to your page and be downloaded and stuff. But it’s important for your own content. If you wanted to get this breakdown of timing details, make sure to add it to your response headers. Here’s a couple examples for popular web-servers of how to add it. We have PHP, JBoss, nginx, and Apache here. Its usually a one liner. But you’re basically just setting a header on the HTTP response.


  So why is it really important? Well, there’s this concept of blocking time. Blocking time is how long the browser is waiting on an open connection if there are other things that are being fetched on it initially. Browsers today, by default, they open up to six to eight to ten, depending on the browser connections to a web server when they’re downloading things from it. If you have twenty images on your CDN, the browser’s only going to open up half of that connections, TCP connections, and it will only send out that maximum number of parallel requests to fetch that data. So if you have 20 images, the browser only opens up six connections, you’re only going to be downloading six things at a time. As soon as one of those images is fetched, the browser kicks off the next thing in the queue and it continues onward. The challenge is that with research timing, you only get the start time and the end time if you don’t have that HTTP header. The resource will actually be reporting how long it was waiting for an open connection, not just how long it was downloading.


  In the example of trying to download twenty images, let’s say it taking a second for every image. Like every image takes exactly one second to fetch, the first six images will report a time of one second. That’s how long it took to go to the server. They didn’t have to wait anything. It took one second to download them from the server. Images number 7-12, will probably report a download time of two seconds because they had to wait a second for the connection, for all the initial ones to be fetched before they can get kicked off. If you don’t have this timing allow origin header, you’re only going to see the total duration of two seconds. So it’s going to look like it took two seconds to download it but half the time was just blocking, waiting on the browser to have an open connection. You don’t get any further details if you don’t have the breakdown of when the fetch actually started, when the DNS actually started and that kind of stuff.


  That’s what blocking time is. Blocking time is the time the browser is waiting for an open connection to be able to download more resources. You can calculate it with some little math right there, for every resource, but that depends on you actually being able to have the breakdown of all the different time stamps for your cross origin resources.


  Okay. Can you see cache hits in resource timing data? Yes and no. There is no flag on the resource timing entry, you know the data structure that we looked at, that says whether it was a cache hit or not. It doesn’t say, you know, cached true or anything like that. This is for privacy concerns. There’s multiple attack vectors, if you know exactly what was in the users cache, I guess. But in practice, you can really guess. If something’s served from the cache, the nice thing about resource timing is you get the how long it took, the total duration. If it was served from your cache, it’s probably going to be loaded really, really quickly.


  In practice, we use a heuristic of if something was fetched within ten milliseconds, it was probably served from the cache. You cannot go to the network and get a response in under ten milliseconds. It could … a cache hit is not necessarily going to be zero milliseconds because it might have to go to memory or to the disk to actually fetch something. But we find that between zero and ten milliseconds, it’s pretty definitively going to be a cache hit. Potentially a little more than that, you might start getting into edge proxies or just a really slow disk taking a while to get it.


Nic: Most grade modems take about twelve milliseconds to get the data out of the modem so that’s kind of a barrier. It’s unlikely to see anything faster than that going past the modem.


Philip: While resource timing doesn’t tell you definitively something was served from the cache. If you want to know your cache hit rate, just look at everything that took under ten, fifteen milliseconds and those are the things that were served from the cache. You could actually calculate your overall cache hit rate for your visitors and JavaScript just by doing this kind of count.


Nic: Yup, question?


Philip: Question?


Female: What internet speed was the load times calculated on?


Nic: So the question is what internet speed was the load times calculated on? These are just aggregated across multiple internet speeds but because we’re looking at things less than ten milliseconds, we’re assuming it’s not actually going off to the internet, it’s staying within the box. Anything more than that, then we’re actually approximating based on the speed of light. Assuming that forty milliseconds means that it’s within the same city or metropolitan region which is most likely an ISB or an edge cache regardless of the internet speed they have.


Female: What can you do with all this data, right? Well we showed you how you can compress it, you can add it to your beacon, you can send it to your background and do some big data analytics on it. As we mentioned before, there’s a lot of bookmarklets. Andy Davis in the front has a bookmarklet that you can click and it will show you the waterfall without opening the deb tools of everything that loaded on the page. Which is great if you don’t want to open the deb tools or the deb tools aren’t available or something that like. You can actually see, basically, the whole waterfall just via bookmarklet. There’s some other cool extensions like there’s the heat map extension that overlays different colors of all the images on your site after you’ve loaded it and like red images, for example, took longer to load and stuff.


  You can actually see how long it took for different resources to load on the page with a heat map of red is bad and other colors are better. Boomerang supports resource timing, so if you’re using Boomerang at all, you can capture it today. It does that impression that we talked about and you can send it back to wherever you want. A bunch of vendors are obviously collecting this data as well.


  Resource timing is pretty well supported today. It’s not as well supported as map timing. There’s about a 70% global market share. The big things that don’t have resource timing are IOS Safari and Mac Safari. Hopefully, at some point, they’ll get that added. Resource timing was added in IE 10 so it’s not in 9 or 8 but it’s in IE 11 and Edge. If you want to capture resource timing in browsers that don’t support it, you can also potentially polyfill it, a little bit. Polyfill is a way of adding in browser support for features that don’t exist in the browser. We have a method that we use in Boomerang to notice all of the resources that are being fetched on the page to automatically capture time stamps for them.


  Basically we use something called a mutation observer. You listen for changes within the dom, changes that are happening, new images that are being inserted, that kind of stuff. If you hook into that and the XML HTTP request object, you can actually get a sense for all the downloads that are happening from your website, track the start and end times, and you can actually make resource time data a waterfall, for example on IOS today.


Nic: One thing to note is that this does not catch anything loaded from CSS like fonts, background images, or other imports. It has to be within the HTML itself.


Female: There’s some sample code in Boomerang, if you’re really interested in doing this. It does add some overhead because you’re profiling the browser essentially via mutations. But if you really want waterfall data for your mobile customers then this is a way of doing it. Okay, so there’s some tips to using resource timing data, as I said before, try to add timing allow origin flag to all over your response headers to make sure that you get the full breakdown of all the time stamps. It’s very important especially for that blocking time. There’s some things that aren’t included in resource timing data, the root page is not part of resource timing data at all. The actual root HTML itself is not in the resource timing data. That’s all measured via navigation timing anyway so you can get the time stamps there. But it’s not included as part of resource timing data. You cannot get the actual HTTP response codes so you cannot know if it’s a 200 or 204, or stuff like that. That data is not exposed in the resource timing data for privacy concerns.


  There’s actually some incompatibility between browsers for errors. If you’re downloading, if you kick off an image and it was a 404 or 500 level error or something like that, some browsers will include that data in resource timing, some won’t. So it’s kind of inconsistent and the time stamps that they log are rather inconsistent too so I think the standard are working towards trying to make sure that there’s more consistency there. But errors are not very well or consistently reported.


Nic: Yeah, also things like …


Philip: …are not very well or consistently reported.


Nic: Yeah, also things like network errors if there was a DCP or DNS error, that’ll just not show up in Resource Timing, mainly because the browser cannot actually read a timing allow origin error to know whether it’s safe to display that information or not.


Philip: We talked about the buffer, how there’s 150 resources in there by default, and you can control it. Make sure if you’re doing that, you’re the only one doing that. For example, if you’re writing a third-party script, you can’t have multiple people actually controlling the buffer at once. It’s one shared buffer. So, if you’re clearing it, nobody else gets the data either. So just something to make sure of. Each iframe on the page has its own Resource Timing data. So if you add an iframe to the page, which loads its own images, those images won’t be part of the parent page’s Resource Timing data. They’ll be part of the frame’s timing data. So in order to get all that, you actually have to traverse all the Iframes within the page, getting all the Iframes’ data, collecting it and bringing it all together.


  Phillip’s looking at the different frames on the page, and from there you can call the same get entries or give entries by type to get their Resource Timing data. But it’s important to know that if you just call it once on the route page, you’re not actually getting all the data within the site. So there’s a bunch of Iframes on the page that didn’t actually download anything, and he’s kind of going through and finding that out.


  One slightly annoying thing is that you can’t always get into all the Iframes’ content. If an iframe was loaded from a cross-origin, and you don’t actually have JavaScript access to be able to go into the iframe, you can’t get its data. This is especially important for things like ads. If you have an ad provider that’s loading data from a different domain, and they add an iframe, they download 100 megabytes worth of images, you can’t actually know that from JavaScript, because you can’t get into that frame to be able to look at this data. So the waterfall provides you with a lot of the — sorry, the Resource Timing data provides you with a lot of the waterfall, but if you’re loading anything from another domain, if there’s iframe loading on another domain, you’re not getting the whole picture. That’s rather unfortunate.


  You will see some weird URLs in Resource Timing data. For example, you might see about_blank, or JavaScript URLs within the Resource Timing data itself. These are mostly due to browser bugs and I think they’ll be getting filtered out in future browser versions. But I think, IE., for example will log JavaScript URLs, just because it wanted to.


  And you might actually see browser extensions fetching data. Unfortunately– I think this is still true– things like Chrome extensions, if they’re kicking off data, they’re going to fetch remote data, that’ll actually show up in the page’s Resource Timing data. And so it will look like your site fetched some random other domain that you have no idea what it’s even talking about, and that’s just because the user had an extension which kicked off a load of something while the page was loading and then you captured it.


  So coming soon. So, all these specs are being worked on, Resource Timing is going to be updated. There’s Resource Timing II. It has a couple of the things we wanted to get into it, which is, for example, the protocol that we talked about earlier, so you know whether it’s speedy or quick or whatever, or H2. Awesomely, it’s going to finally add byte sizes, so, how much actual content was delivered over the network. There’s three different things there: there’s transfer size, which is the total amount of bytes transferred across the network, there’s the decoded body size, which is the actual size of the content, and there’s encoded body size, which is, if it was gzipped, for example, the size of the content encoded gzipped. So these are being added. I think they’re in Firefox current and Chrome… Canary or something. So, you could actually get total page weight, how much content was actually downloaded to build your page, once this is available.


Nic: So if you have Firefox, you can open it up right now and check performance to getEntries, and you’ll see all of these times, these byte sizes, in there.


Philip: How are we doing for time?


Nic: We have 25 minutes.


Philip: 25 minutes. Okay. Do you want me to do this? Or do you want to?


Nic: Go ahead.


Philip: Okay. So, User Timing. By the way, any other questions about resource timing at all? Okay. User Timing: so User Timing is another browser API. And it allows you to measure important events within the page, and important durations, but it’s from your perspective. It’s from the developers perspective. The browser isn’t going to do anything with User Timing unless you tell it to.


  User Timing will help you keep track of time stamps, which it calls marks, and durations, how long something took, which is a measure. This is a screen shot of Chrome DevTools, and you can see, actually, there’s a User Timing row, and there’s four different measures logged, four different things that took some duration. And you’re basically annotating the timeline with things that were important to you, important life-cycle events within your own app.


  Now, you could have done this before. You can keep track of time stamps however you wanted to, even just with the data object, even though we told you about all the different problems with the data object. But let’s say you wanted to measure how long it took for you to calculate the number of widget somethings. You could keep track of the start time and the end time and get the duration from that. But User Timing’s actually a better way of doing this.


  So, the two core concepts of User Timing are mark, which is just a time stamp, it’s just a period in time, and measure, which is how long something took. And you’ll combine the two. So, if you log two marks, for example, you can create a measure of how long the difference was between the two.


  It’ll make a little more sense if we just do an example. You know, before we were saying you could do, for our start equals new date, get time.


Nic: You don’t need the brands.


Philip: No. And then start is ginormous, 1.4 trillion, timestamp. But we already talked about all the problems with using the data object. User Timing, on the other hand: let’s say you’re in your code, you want to measure the beginning of an event. You do performance.mark, Start. Now you don’t get anything back from it. But it put this data into the performance timeline. The same place that the Resource Timing data went into, this data also goes into. And you can get it by doing getEntries by type. And the type in this case would be mark. In this case, we see the one mark we already put there, the name of Start, and it happened at, in this case, 25,000 seconds since we loaded the Google Slides.


  You can add another mark, and you can get the same data. In this case you’ll see both of them here, and they happen at different timestamps. So that’s a mark. Let’s add an end mark, too, just for consistency.


  So a mark is just a point in time, it’s a timestamp. A measure is the distance between two marks. It’s a duration. And you log a measure via a very simple API as well. Now by default, the API parameters are the name of the measure, so we’ll call it My Measure or something, and then you can specify the start and end marks. So I want to calculate the difference between start and end. Nothing’s returned, but the data was put into the performance timeline. I’m going to get, instead of a mark, my measures now. So there’s one entry. The start time was a start time of start, and the duration was the difference between start and end. In this case, it took me about 16 seconds to do mark start and mark end. So we had a duration of 16,305.


  So that’s how simple marks and measures are. That’s it. That’s most of it right there.


Nic: Do you want to talk about default values?


Philip: Oh, yeah. For measure 2, there are some default values you can use. If you only specify, for example, the name, it’ll take navigation start as the default value for start. So, it’ll basically base your duration from the beginning of the navigation until now. And end is now? Yeah. If you don’t specify a value for end, it’ll take now as that. So you don’t actually have to do an end mark, you can pick which end mark you want, or you can pick now.


Nic: Things like and response start work.


Philip: You could also pick other mark names that are in navigation timing as a start. So if you cared to…


Philip: … Mark names that are in navigation timing as it starts. If you cared to measure the difference between response start and now, you could specify that as the start value and it’ll go into the navigation timing data and use that as the basis. It gives you some flexibility from what you want to measure from and to, and it puts it into that resource timing. Sorry, into the performance timeline. You can clear these marks, you can clear the measures, you can get them out of the array, and that kind of stuff. That’s the majority of the actual API.


  Here’s an example of basically what we just did and measuring. Why do you want to use this versus just doing it on your own? Well first thing is that it puts this data into the performance timeline. It puts it into a standard place where all the other performance data is being logged. It’s right next to your resources and other things, so if you’re looking at the performance timeline sequentially, you can see your data there too. It uses the down hi res time stamp by default so you get all the high resolution, monotonically non-decreasing time stamps, so you’re not building with the data object. One of the most important things is that it gives a standard way for other people to look at the data. Browser dev tools, third party analytics scripts, et cetera, they all know where to look for the important events within your site.


  If you’re a developer and you are marketing important time stamps or important durations, you only have to do it once and you can swap in and out different third party vendors. You can look at Google Analytics. You can look at your dev tools and they’ll all be reporting the same data: the same time stamps that you logged once. You only have to actually instrument your application one time. More and more vendors are adding support for looking at the user timing data. User timing supported pretty well as well. It’s not supported in Safari yet. It started in IE 10, I believe.


  You can polyfill it which means you can add support to it for old browsers all the way back to IE 6. It’s a relatively easy API. I have a polyfill available, usertiming.js and you can basically add it to your page and if user timing is not supported, it adds support for all those APIs that we saw, so performance.mark, performance.measure, and that kind of thing.


Nic: If you’re using a user timing polyfill and a results timing polyfill, just make sure they work well together. Sometimes they over write the same object.


Philip: Similar to resource timing data, you might want to compress it instead of just stringifying the whole thing. We have a couple different ways that we’ve built to compress this data automatically and get it down to a more reasonable size if you want to send it to your back end. You can check out this repo if you want to do that compress it and decompress it. One of the things that we talked about with user timing is that you get the marks within dev tools. This is an example of edge, I believe. You can see the user measure …


  It’s kind of hard to see, but the highlighted line is basically when somebody was doing a measure within the browser. This is a web page test. A web page test will show you different user marks that you’re logging within your site. Again, a standard way to log important events, and then all these different vendors and other analytics tools can actually still hook into those. You don’t have to tell about them specifically.


Nic: Yeah a lot of people do this like they put a timing mark before or after an ad block to just measure how long their ads are taking to load. That’s because you don’t necessarily know which domain your ad is coming from but you know you can specify a pre defined mark.


Nic: Yeah. We’re going to go very quickly over some other useful APIs that we use for measuring. One of them is the page visibility API. This is just something that tells you whether your page is currently visible or not. There are three states, four states really, but three that we really care about. There’s the hidden, visible, and pre-render state. Hidden happens any time your page is currently not visible to the user. It might be minimized, maybe in a background tab, maybe in a window that’s covered by another window. All of those cases, the visibility state is set to hidden.


  What you can do with that when the visibility state is hidden, is disable the expensive things like if there’s a lot of animations happening, you can stop animations because the user’s not looking at it. You can slow down the rate of pulling the server if you’re doing that to figure out if there’s new content. For example, I think Gmail does this: if the tab is visible, it’ll show you new mail at a higher rate than if the tab is invisible. If it’s visible obviously, you can do much more.


  The pre-render state is interesting. If the browser downloaded the page and pre-rendered it but hasn’t made it visible yet, that’s when it’ll be in the pre-render state. This is useful for your analytics tools because you don’t want to mismeasure that as a page view because the user hasn’t viewed that yet. Let’s say the page was pre-rendered, you look at the pre-render state before you send a beacon for page view measures. When it moves from pre-render to either visible or hidden, that’s when the user actually requested the page.


  In Boomerang, what we also do is look at the time between the user requesting the page to becoming visible or hidden, because that’s what the user actually experienced versus the time when navigation happened and pre-render happened, which is when the browser downloaded the content. There might be a difference between the time it took to pre-render versus the time it took to make the page available to the user.


  Finally, there’s the unloaded state which really happens once you get into the unload event. It typically means you need to stop doing everything now because all bets are off. Very few things will work in the unload event. The way to use it, there are two different things you could use. There’s a documented visibility state. There’s actually also something called document.hidden which we haven’t mentioned yet, which is either true or false. That doesn’t tell you things like pre-render. Documented visibility state will be one of these four states.


Philip: I think document.hidden is going to be deprecated.


Nic: Yeah.


Nic: Then there’s an event that’s invisibility change. If you listen to that event, every time the page either goes from pre-render to hidden or visible, or visible to hidden, hidden to visible. It’ll file, and inside that function, inside that event handler, you can get documented visibility state. We do this, for example, to tell how long the page was actually visible to the user when we send a beacon. Every time the visibility state changes when it’s hidden, we will stop measuring that time. We can measure total eyes on time versus eyes off time for that page. It doesn’t tell you that the page was visible but the user was away from their computer. That’s not something we can measure yet, but we are working on it. Support is pretty good.


  All browsers, except Opera Mini which doesn’t actually run on the device, can tell you if things are visible or not. Pre-render is only supported in browsers that actually support pre-render, which I believe are IE 11 or 10 and above, Chrome, and Edge. Anyone know if Firefox supports pre-render? I don’t think it does.


Audience member: Opera …


Nic: Yeah. Opera does. Anything Chrome supports, Opera supports because Opera uses Chrome’s engine. I always forget about Opera. Bruce is going to kill me. I’ve told him.


Audience member: It’s the European browser, I should say.


Nic: It’s the European browser. It’s a cool browser. Another cool API is request animation frame. If you worked in JavaScript recently, you might be familiar with that. Is anyone here familiar with request animation frame? All right. A few people. Anyone familiar with set time out, or set interval? All right. Request animation frame is kind of like set time out, set interval, except it waits until the browser is ready to render something. If the browser is busy, it’s not actually rendering anything, you’re not going to get a request animation frame. It’s good because it lets you wait until the browser is not too busy to do something.


  We can also use it to measure things like frame rate. If request animation frame is firing at a consistent rate, that means your frames are being rendered smoothly. In our case, in this example, we’re just looking at it to determine when to update something or not. If you’re doing animations, if you want your animations to be smooth, use request animation frame and every time a frame fires, you calculate how far you need to animate. If you’re doing CSS animations that all happens automatically for you.


  That’s also supported fairly well, just as well as page visibility. Got a 90% browser usage. There are still some browsers that have this prefix. You have web kit … I don’t think web kit anymore but never mind. There might be users that have a prefixed version, so web kit request animation frame of MS request animation frame.


Nic: Page load milestones … We mentioned all the navigation timing milestones, but there’s rarely milestones that you, as a developer, care about and that the user might care about. The first byte is once the browser actually starts receiving data, which is when you can start doing interesting things with the page. That’s, the navigation timings response start tells you that point of time. You know, you can’t do anything before that. Anything before that, you complain to your back end guy or your ops guys or something, saying, “Hey, there’s a network problem.” Anything after that, it’s your problem.


  The on-load time is when all statically loaded content and any content that was loaded dynamically before the on-load event started. If you have a JavaScript that’s adding other JavaScript images and that JavaScript runs before the on-load event, like in the dom complete event or something, that is going to be counted towards the on-load time.


  Anything that’s injected into the page during the on-load event is going to fire … The on-load event is obviously going to fire before that happens, or anything that’s loaded in an iframe whose on-load event is already fired. That’s something we’ll talk about a little later, but certainly, if you have an iframe, once its on-load event fires, that iframe is no longer going to block the current document’s on-load event. Anything loaded in that iframe after that Iframes on-load event might load after the … will not block this page’s live on-load event.


  Off that, we’ll go into the fully loaded section, which is when we say all of the content required for this page is loaded. This is more important when you deal with single page apps, which might download something like Angular JS right up front, statically, in the page, and then Angular JS will go out and fetch content, JSON, or other structure needed for the page, depending on the current view. That’s where the difference between on-load and fully loaded are going to come into play.


  It’s important to know that the visual experience that you care about, that your user cares about, are not the network timings. All of the network timings that we get from navigation timing are interesting in terms of looking at infrastructure, decided what needs to be fixed there, or whether your paid structure needs to be changed. It does not tell you user experience. It doesn’t tell you whether your users are happy or sad with that experience.


  What can we tell about visual experience? There are some milestones with visual experience that we care about. There is the first paint, which really may not be very useful. We’ll talk about that in a bit. There’s the first content full paint, which is, something was on-screen. Whether that this was useful or not, we don’t know. Then, there’s the first meaningful paint, when it’s something the user actually cares about is on-screen. Then, there’s visually complete, which means everything that you intended to put on screen was available, whether the user cares about it or not.


  The metrics that we can measure visual progress, how long does it take for different things to be available on-screen, and speed index, which is, at what point of time does a certain amount of the screen available. When we talk about visual experience, what was the first thing the user saw? It’s first paint. Most likely, painting the screen white. Does the user care? If they see this for too long, they’ll probably get frustrated.


  This can happen between this time, any time from the first byte until on-load event. Remember when I mentioned that the unload event of the previous page files when the first byte is available? What also happens is the browser then takes away the visual of the previous page. Until the first byte comes in, the user’s still looking at the previous page. When the first byte comes in, the browser clears the page, so now you’ve got a white screen. Unless you have something to display at that point of time, the user’s going to be staring at that white screen for a while.


  Here’s a timeline, taken from first feed. It’s looking at first paint, which is not an industry standard. Different browsers have a different idea of it, so in case of Internet Explorer, we know that performance of timing that MS first paint, if you have IE, you can look at that right now. In Chrome, we already told you about Chrome load times, which has first paint time. They’re measured in different units. The first is in milliseconds. The second is in, I think it’s seconds, not microseconds, although they might have changed it to use higher time stamp.


  First view is no content. Then, you might have some content, full content, but the user doesn’t necessarily care about, in this case, just the header, but it’s kind of indicating to the user that things are happening. Your browser’s not hung, things are not broken. If it stays on this too long, then the user might think things are broken.


  First meaningful paint is when your primary content is available. Your user wanted to read an article, that article’s text is starting to be available, the images are starting to be available. That’s meaningful paint. Visually complete is, all the stuff on your page is now available, all your ads, and everything that you consider content is available. It’s not the same as the on-load event, because you could actually load content after the on-load event and make that visible. We consider visually complete when the page hasn’t changed in a while.


  Besides first paint, none of these other metrics are available, that we mentioned. There’s no event for content, full paint, there’s no event for meaningful paint, there’s no event for visually complete. Although, there are working group projects. Performance working group is looking at some projects to get these things into the spec. Some of the things we’re looking at is first white, or really, first non-background paint. If you have a background color in your CSS, when was the first paint that was different from that background color? When was the first non-blank paint? The first thing, after you clear the screen, when was that? Those 2 metrics will be interesting to tell when did the user actually think something was happening with the screen.


  You want to talk about PW?


Philip: Yeah. Paul Irish recently added this open-source script, PW Metrics, that’s actually trying to calculate all of these things, through some heuristics. It’s an interesting way of looking at what we’re talking about, from the first paint, to the first meaningful paint, first content full paint. You can kind of get a timeline from what the working group’s thinking as how long it takes for your page or your site. You can run it against any URL, it’s a command line tool.


Nic: Then, we talked about visual progress, how fast is content being drawn on-screen? This is something you can get with a webpage test. For example, it’s the speed index. I’m not going to explain how Pat Cath-o-lit’s speed index, he’s got a very long blog post on that. Essentially, it tells you how much of the screen is visually available at different points of time, rather, at what point of time is the page complete. Everything above the fold. That’s a good indication, to compare your before and after, if you’re making changes to your site, has your speed index improved or not, or to compare yourself with a competitor. What are your speed index values?


Philip: For speed index, the lower is better. It’s a number, it’s the average time at which the content on the page was displayed. Again, it’s kind of a complex calculation, hard to explain, but it’s actually a pretty good metric for measuring and comparing time over time.


Nic: There are a few downsides. It’s not very well understood. It’s hard enough explaining these things to non-technical people. Speed index is hard to explain to technical people, as well. It takes a while to read through and understand the math before you can explain it to somebody else. It can only be captured accurately in a lab. Running webpage test is a good way to measure your speed index. It’s also a good way to compare your before and after, because conditions are non-changing, but measuring it from your user’s point of view is harder.


  There is something called Rom Speed Index, which Pat has also built a JavaScript library to measure speed index. It’s not as complete as doing it in a synthetic way, because you cannot just say how many pixels on the screen have been drawn. It needs better support for Iframes and SPGs and stuff. Actually, what it does is, it looks at results timing data, to see when images above the fold are completed loading, and if there are any dibs, did they have a font or a background image? When did those things complete loading? That tells you when parts above the fold were available to be rendered or visible to the user. That’s when we say things are visually complete above the fold.


  It’s somewhat accurate. It still needs a lot of development and a lot of real-world research, but we’ll get there, eventually. If anybody wants to work on that project, get in touch with us. We’ll figure something out.


Philip: So I think we’ll take a break now … 30 minutes


Philip: One thing we noticed about … what we’ve seen so far with navigation timing, resource timing is that it tells you a lot about network activity. Navigation timing in particular ends when the on-load event fires. And that’s typically when most single page applications start. So it’s not very useful for single page apps. Most frameworks that you use tend to have their own events saying when something is requested and when something is being fetched from the network


  We looked at all of these and we found that they were also lacking things like resource timing lack, and telling you everything. They’re telling you when things came off the network but they’re not telling you anything about visual experience. So what we tried to do with measuring performance of single page apps is go all the way down to the visual experience and measure what the user is actually experiencing with your app rather than what the network or the browser experiences


  Does everyone have the slides in front of them? We can jump through some of the text only content while we wait for these things to come up.


  Do you want to go through it or should I?


Nic: How long do you think it will be?


Philip: I can talk about the challenges of measuring single frame apps



So those of you who have built single page apps, do you support both full page loads as well as soft navigation’s? So, we call them hard navigation’s and soft navigation’s. Or is it only soft?


  We try to measure both, but the experience is typically different for soft vs. hard. Also, search engines tend to notice the hard navigation’s while users tend to notice the soft navigation’s. Synthetic monitoring tools tend to notice all the hard navigation’s, so they don’t usually measure the same things that end users do.


Philip: So the questions is, what’s the impact of using the performance on performance itself. And especially if you have to do it many times, like if you’re doing it for fifteen ads or something. So the point of using performance mark vs. new data is that there is much less impact. In fact, it’s built into the browser as part of timeline there is no impact except the memory for using that mark. And that’s fairly minimal at this point.


Philip: Yeah so all JavaScript is typically blocking the setting, like this is just writing to memory. So it’s not going to be more than … if you measure how long it takes to make a performance mark itself, it’s less than a millisecond so it shows up in the microsecond range.


Philip: The question is what does window . frames get you. Window . frames gets you all of the objects available within this current window. So it would also get you the current window – the zero will be the current window. It’ll get you all Iframes.


  Now if your page is a frame page – like it’s frame set – it will get you all of those frames. So it’s essentially of all those only available within the current window. So if there are nested Iframes, it doesn’t get you those. You have to go into one level and then look at window . frames.


Philip: So the question was: does an Iframe effect your load time? The answer is yes. So if your Iframe is loaded statically, then anything within the Iframe will effect your main pages load time. If your Iframe is added dynamically but added before on-load, then it will. If the Iframe is loaded dynamically and the content within the Iframe is loaded dynamically after the Iframes on load, then it will effect your bandwidth but not your page load time. So it will use up network resources to download and therefore it might slowdown downloads, but it won’t actually effect the load time.


  So what you want is an iframe that you basically let the Iframes on load a file and then load content in the iframe after that. And in the references of our slides, there is a link to how to do that.


Philip: So what I typically do is I cradle them and I look at location inside the iframe. So if you go look at window . frames I crate through that collection. Each of them will give you a frame and you look at frame . window . element . atrif


  Does anyone know if I’m right or wrong? I just said that from memory so I’m not sure if that’s right. It’s either window element or document element . window. It’s something like that.


  All right, do you want to go ahead?


Nic: All right, so we’re going to talk about single page apps. Single page apps are different than how we traditionally built website. Single page apps, in general, what you’re doing is you’re loading the core framework. The framework then goes and decides what views it needs to render. So when you’re looking at an angular app, the angular framework itself is loaded. Angular looks at what URL you went to, decides that there’s some templates it needs, some images, and all that kind of stuff. It goes and fetches all this data dynamically and then builds the page from there. And then when you’re navigating, when you’re clicking on different links within a single page app, you’re not actually doing the navigation anymore. The framework itself is deciding that there’s a different view. I’m gonna blank out the old view, bring in the new view, and it just kind of dynamically swaps in the content that you’re looking at. We talked about some…




Philip: The content that you’re looking at. We talked about some of the more popular frameworks like Angular, Ember, React, Backbone. You could build your own, all that kind of stuff. Single page apps perform a lot differently than a traditional website and you have to measure them a lot differently.


  So in a single page app there’s two different types of navigations. There’s the hard navigation and a soft navigation. The hard navigation is the first time a visitor comes to your website. You know whether they come from a search result or whether they type in the URL in the address bar that’s a hard navigation.   Hard navigation has to do a lot of work it has to get all the core HTML for the page. The JavaScript, the CSS, the framework itself, so like AngularJS and then the framework decides it also has to get the view. So what you’re actually looking at. It also loads in all that dynamic content. So all that together is the hard navigation.


  A soft navigation is anything after that. So anytime somebody is interacting within the site, they’re going between pages within the site and the browser again is not really actually unloading the page and reloading the next page. It’s dynamically flopping in this content. That’s a soft navigation. It doesn’t have to re-fetch Angular JS every time so it’s actually a lot cheaper to do, it’s a lot more performing and quicker.


  In general most frameworks allow you to visit a page either from a hard navigation or a soft navigation. So thinking about like your home page, for example. If somebody types in your home page URL that’s a hard navigation. They could navigate away to a product page and then back to the home page and that’s a soft navigation to your home page. So you really need to organize the data that you’re looking at, because they have vastly different performance characteristics because a hard navigation is doing so much more. So most of your URLs could probably be visited either via a hard or a soft navigation.


  So we’re gonna talk about the three challenges of measuring single page apps and why you can’t just look at navigation timing. Why you can’t just look at the time-stamps. The first challenge of single page apps is that the onload event doesn’t really matter anymore. You know, in a traditional website the onload with navigation timing that we looked at is when all the static content on the page has been downloaded. When all the resources that were there when the page first started, when they’ve been all fetched. That’s when the onload event fires. This is what we would traditionally consider a page load complete. This is just a sample, a blog, waterfall of all the resources they all get fetched you know, in a serial manner and the page load is at the very end of this. That’s when the onload event fires, that’s when navigation timings, load events start, that’s the time-stamp for that. Basically after everything is completed right.


  Well as we’ve said it’s different for a single page app. Generally with a single page app framework you’re downloading the framework JavaScript itself. So like Angular JS you’re going to fetch and at that point often the onload event fires because that was the only thing that was statically included in the page. However, angular wakes up and is like oh hey I got to go get the view, I got to go get the templates for the HTML and all the images that you want to look at. You know I got to actually render what you’re looking at. So in that way the onload event it’s somewhere in the middle, it’s not when the user sees a visually complete page it’s just when the framework itself had been fetched.


  This is a single page app waterfall. So this is the angular demo app. Right in the middle of the page is when the onload event fired. Everything to the left of it is kind of like the core content that needed to be fetched once for the page. So HTML, I think there’s a couple JavaScripts, a couple CSS and in fact the longest resource there is AngularJS itself. That was fetched the page fired the onload event this is when we would traditionally stop monitoring. Everything after that is the actually content that we’re gonna show. In this case I think it’s a listing of phones. Popular phones, android phones. All of the things after the onload event are the images. So like the actual content you came there for. So if you’re just measuring to the onload event you’re not really going to see the full picture.


  In fact this is a visual timeline. We use web page tests to get a video of the page load process. Now onload fired at about 1.2 seconds. But if you’re looking at the timeline it’s a little hard to read but the first two frames are from 0 seconds to 1.6 seconds and there’s nothing on there. I mean there’s maybe a little bit of text in the upper left corner but would you consider the second image complete? Probably not right there’s barely anything there. But that’s when the onload event was in fact, 200 milliseconds or 300 milliseconds before that. It’s not until 1.7 seconds that some text starts coming in. 1.9 seconds until the actual images have been all fully loaded. So we don’t want to measure onload for single page apps. It’s just not relevant anymore.


  That’s the first challenge. The second challenge is that soft navigations are not real navigations. Remember a soft navigation is anything after the first navigation. The single page app framework will go fetch all the dynamic content that it needs and replace the view with the new URL you’re going to. In the browser, this is great for performance because the browser doesn’t have to re-fetch the header and the JavaScript and you know AngularJS and stuff. The problem is that all the browser events are geared towards a single load, like, navigation timing has the onload event and that’s it. Any time you’re going to navigate within the page after that those time stamps don’t get updated. That was the one load you had, nothing else gets emitted from the browser. The browser doesn’t track any time stamps after the load event. So every time your navigating within the site you’re actually not getting any new data from the browser.


  The third challenge with single page apps is that the browser won’t tell you when all the downloads have completed so after the onload event, remember the onload event is when all the static content on the page has been downloaded, well we actually want to know when all the images have been downloaded. The browser won’t tell us that at all. There’s no additional onload event that you can hook into. There’s no way to know that there’s still something being fetched. You can’t actually ask the browser, do you still have a download going on. There’s no way of doing that.


Nic: They want an on more loaded event after the onload event.


Philip: Is there something else on the network? And if you don’t know if whether the network is still being used. You can’t actually say or know definitively that the navigation has completed. You know in single, in a soft navigation, a hard navigation it can fetch all these different things. It can fetch HTML templates, CSS, JavaScript, images and these all contribute to the visual completion of the page. If you’re not measuring these you’re really not measuring the user experience.


  So for single page apps we need to figure out a couple things. We need to figure out, like with a traditional website what’s the starting event and what’s the ending event? In a traditional website the starting of the navigation is when somebody’s clicking on something and the end is when all the resources have been fetched, the onload event. We need to figure out the same start and end points for single page app.


  The good thing is we can find those events from the browser. So for a hard navigation, for when somebody is first coming to your website. The starting event is when they actually navigated, right. When they clicked on a link, when they typed in the URL in the address bar. So we can actually use navigation start as the beginning event for hard navigation.


  For soft navigation when somebody is going within your site, we need to figure out what was the trigger. What caused the actual navigation to happen, and we only want to track when the users view is gonna significantly change too. We don’t want to track every single click. The user could just be scrolling, they could accidentally click on some text, they could be like click reading. I don’t know if people do that, like drag across the text, like highlight it. Those aren’t really navigations. You know those are in page interactions but they’re not a significant event from the navigations point of view.


Nic: You also have things like an auto-complete search box where the user types in something but you don’t want to measure every single key pressed there. Even though that results in a network activity. That’s not significant, competent of user experience. So we want to measure things that are meaningful to the user, rather then everything on the network.


Philip: Right. But there are other things we can listen too. We can listen to the history changing. So there’s a window.history object that frameworks look into. That’s when the URL changes in the address bar even though you haven’t actually navigated, you can actually fake the URL changing. We can listen to those events. If the URL is changing it’s very likely that the view is changing. The frameworks themselves emit some events when a significant thing is happening. We can also just pay attention to clicks and XHRs and kind of notice what happens afterward. We can take all this information together and use that to decide when to start listening for a navigation. Again the browser doesn’t tell us any of this, there’s no standard soft navigation starting event. So we have to kind of come up with our own.


  The frameworks themselves they do, you can hook into different things and listen for when the view is about to change, when the route is about to change. You know, when a navigation is happening. So we have some pretty easy ways with the built-in frameworks. But we can also just listen for clicks and other interactions on the page and see what happens after that. As he said you know, like doing your auto-complete is not really navigation but if the user is clicking on a homepage element or a link to go to another part of the site, you can actually watch what happens, you know. Pay attention to the DOMs, see how it’s changing. Pay attention to the network, see if we’re fetching a lot of data and if we are and that results in like a visual change to the page, a significant visual change to the page, you can treat it as the beginning of an interaction for a single page app.


  So we have all these events to start with, these are all the beginnings of a soft page, soft navigation.


Philip: Beginnings of a soft navigation. We just have to figure out what the end is. When do we stop listing? After a user clicks on a link, what’s the end time? Remember, for a traditional website we would consider the end to being when all the resources have been fetched, when all the networking activity is complete. As we said with the challenges, we don’t actually know that for a soft navigation. We can’t use on load because that only tracks the resources once when the page first loaded not for single page apps. So we need to come up with our own on load event for single page apps.


  Basically what we want to do is we want to be able to monitor the networking activity on our own, know if there is any network fetches that are still outstanding, and if there are we’re going to extend the length of the navigation.


  We’ve found a couple ways of doing this. Two things that you need to pay attention to know about all the networking activity on the page. One are programmatic ways of downloading data, so using an XHR is a programmatic way of fetching data. The other way is implicit if you’re going to change the DOM and you’re going to insert an image into the DOM.


  The browser is going to go look at that and notice the image you don’t have and go fetch it from the network, so you need to pay attention to both. Both what the JavaScript is doing to fetch resources as well as what the HTML is doing to fetch resources.


  The good thing is we can monitor both pretty easily, so XHR’s, you can proxy the XHR object basically replace it with your own. When anybody calls it, when they call open on the XHR or send on the XHR, you can log those time stamps and you can know whether there is XHR’s outstanding on the network and how long it’s taking for them to download.


  We have some sample code, we don’t need to go over it here, but you can easily do this, you can easily proxy the object and know when all the XHR activity on the page is happening … You also have to know about the HTML. If the view is changing, if all the content is being swapped out and a new image is being inserted you want to know what images are being inserted, when they completed downloading.


  There’s other things that are important, too. JavaScript and CSS. All those things are important. If they change, if they get added to the page they’re going to contribute to the visual completion of the page.


  The way that we do this is with something called a mutation observer. Mutation observer is a very lightweight interface that you can use to monitor changes to the DOM. This is anytime the HTML changes within the page dynamically you get notified of it. You can listen for certain types of elements that get changed, you can listen for images to get added, script tags to get added, that kind of thing.


  Every time you get notified of this you can actually hook into it and set a event handler for when that image or that script of whatever eventually loads. Then you know that there is activity outstanding on the network. We kind of talked about this before when we were talking about rebuilding a waterfall for IOS. You can use mutation observer to know about all the outstanding network activity, and, most importantly, you can know whether there is anything still being downloaded. Remember, we’re trying to replace or build our own version of on load for single page apps.


  Again, we have sample code for this, we don’t need to go into it here, but if you’re interested in doing this it’s fairly straightforward. We don’t want to listen for everything on the page, but we do want to listen for things with a source or HREF attribute, so images, script tags, link tags, I-frames, video potentially, that kind of thing. You want to make sure that you’re only looking at images for example that are actually going to be fetched so if they’re already within the page and the image is re-added it’s not going to have another on load event because it does not need to go to the network. You can look at things like is the natural width already defined? If so the browser already has it in it’s in memory cache. There’s a couple other caveats for things that you want to download or that you want to pay attention to being downloaded.


  Once you get all this data you can actually reconstruct a waterfall and you can reconstruct knowing whether or not the browser has completed fetching all the data that’s outstanding.


  We can’t actually use … resource timing is kind of a similar thing of being able to monitor all the resources on the page. Unfortunately, you can’t use resource timing for this specific activity, because resource timing events only come after the download has completed. The data only gets added to the performance timeline once it’s complete. We don’t know if there’s anything outstanding on the network, if the browser is trying to fetch something. Without that knowledge, we can’t know whether or not to extend how long the navigation is happening.


  When you put those two together you get a start time and an end time. You know the duration of a soft navigation. The user’s clicked something, the routes changing, you start monitoring, you wait for all the resources to be fetched, and that’s the end time. That’s your on load event for soft navs. Boomerang will do that automatically but it’s fairly easy to do as long as you know what you have to monitor and how to calculate it.


  Single page apps also have a slightly different definition of a front end time versus a back end time. Remember, this is the navigation timing timeline. In general, when people talk about the back end time, how long it takes your servers to respond, they’re talking about from the beginning of the request to the first bytes that are coming back. That’s how long it took for the back end to come. The front end time is everything after that. That’s when you’re downloading images and CSS and that kind of stuff. That’s when the browser is doing it’s other work up until log in.


  For single page apps the same definition isn’t exactly the same because you could actually have multiple things that are going on the network that are critical. For a soft navigation, what if you have to send out two or three XHR’s to bring in the data that you need so you know how to build the page, how to build the images, and stuff like that.


  The way that we define back end for SPA is any time period where you’re waiting for a critical resource and we define that as an XHR or script, and the front end time is all the opposite time of that. Any time you’re not waiting on an XHR or a script, we’re defining that as front end time. When the browser is doing work getting images or CSS and stuff like that.


  Any of you guys using accelerated mobile pages, AMP pages at all for your websites? You know about AMP? One, a couple?


  AMP is a similar … it’s a project started by the Google guys to make really performant pages, especially for mobile devices. The way that they do this is they really restrict the amount of content you can put in them.


  Google promotes sites that are publishing AMP pages, and you can think of an AMP page like if you have a blog, your content, you will also publish an AMP page that has the same content, but you can’t put some things in it. The restrictions are you can’t put, for example, third party or first party JavaScript in it. There’s a couple other things as well. This helps it make it more performant by placing restrictions on what you can have in your page. It ensures that the view itself will load pretty quickly, but those restrictions mean you can’t actually measure how visitors are viewing your AMP pages at all.


  I won’t go into details since you guys aren’t using it very much, but the nice thing is there are some ways in AMP … we’re actually going to talk about this in a talk on Thursday … There’s some mechanism within AMP, if you do want to measure how long your AMP pages are taking to load. There’s some tags that you can use within the page to send out data, to beacon data back to yourself. With the time stamps that you would get from navigation timing and stuff like that. Both an AMP pixel tab and an AMP analytics tab. They’re pretty easy to configure. Even though you can’t include third party JavaScript or even your own JavaScript in AMP pages, you can use these tags to still get data out of them.


Nic: Before we move on, are there any questions about single page apps or AMP?


  We’re going to talk about measuring continuity. So far we’ve talked about measuring everything up to navigation complete whether it’s a single page app or a full page load. We measure whether users bounce just knowing whether they’ve gone onto another page or not. We know those things. The bulk of user interaction happens after on load and before the user leaves the page. They might read content, they might click on things, they might select text. They’re interacting with your content, but you don’t know about that because we’re not measuring that.


  What we really want to know is which of these continuous variables that the user … Things that the user is doing within your page. Which of those can you measure? Which of those are impactful to user experience and how can we measure them?


  If we look at the developer tools timeline, this is Chrome dev tools. This is the timeline tab. Is everyone familiar with the timeline tab? If you’re not, go ahead, open dev tools and switch the timeline tab and then reload your page. Have a look at how the different things that happen, the different kind of activity. It tells you how much of your CPU is being used, how many objects are being created, how much network bandwidth is being used, where time is being spent. Is time being spent in rendering, is it being spent, is it being spent in downloading, is it being spent in executing JavaScript. All of those thing are happening over the course of …


Philip: Script, so all of those things are happening over the course of a user using a page. That’s after the load has happened or even while the page is loading. All the way up to when they leave the page. These are things that we’re not actually collecting.


  There’s this famous, well I don’t know how famous it is, but there’s this quote from the Performance APIs, Security and Privacy, that says, “Just because something is possible to measure doesn’t mean it’s a good idea to measure it or even desirable to expose it to developers because it could compromise the users privacy”. A good case is RAM, you don’t necessarily tell any third party developer how much a user visiting your site has or how much of your CPU is being used because they can use that to make guesses about the user’s system, then do all kinds of third party attacks. We want to limit how much we can expose by APIs.


  At the same time, as the site owner you want to know is my site actually causing the user a frustrating experience by being non-responsive by creating to much Jank, etc. We zoom into one of these things in the network panel, it tells us the things that are available to developers who have a browser and a testing page by themselves but not available to developers who want to see how an end user experience is. In this case, the frames second CPU and network size, 4 different measures in that that we can tell in our own browsers but we can’t necessarily tell on our end users browsers. Some of the things we wanted to do is try and figure out other ways we can proxy this, a guess at what these metrics are for real users. For frames per second, a very simple measure we took was looking at request animation frame and seeing how frequently that request animation frames files.


  If everything is smooth, the request animation frame should fire very regularly and you can time the difference between subsequent calls. If that’s constant that means your frame rate is smooth, if that’s not constant that means the browsers busy doing something else and cannot actually render things. If you measure request animation frame over time, typically any frame that takes more than 16.6 milliseconds leads to under 60 frames per second which causes Jank. Measuring this, sending this data back tells you whether your users are experiencing Jank or not. Similarly we can do that for video. Video actually has APIs that already provide this. It was the decoded frame content and the painted frames. It’s not very visible in the frames but if you have the slides downloaded have a look at that.


  It tells you for video what’s the frame rate that’s coming through and how often is it being painted to screen. In terms of CPU busy, it’s a little harder but you can take the concept that the inverse of your frame rate is typically how busy your CPU is. It can’t actually render anything that’s probably because it’s busy doing something else. What we do is run a function at a regular interval, either request animation frame or set interval. Using measure if you expect your set interval to run every 100 milliseconds, does it in fact run every 100 milliseconds.


  There are some caveats with that. You don’t necessarily want to do it on a mobile device, which is not doing anything else. Because that has to wake up every 100 milliseconds just to run your measurement code and then go back to sleep. It makes it harder for the browser to conserve battery in those cases. If your actually measuring passively, which we can do in this case. There’s various things that can cause the browser to go busy and delay set intervals like layout, downloading content of the network, or running under JavaScript. Since JavaScript was single threaded, any JavaScript you run is going to block some other JavaScript from running. Here’s some code to measure CPU page busy. In this case we’re using set interval and a function that collects the time stamp. Later on, we check to see whether the rolling interval is less than the actual interval that we saw. In that case we know that the CPUs busy.


  In trying to guess how much of the network is utilized. You can guess at it with resource timing, just by seeing how many resources are being loaded and peril. If a lot of resources start but don’t end you can assume they’re loading in peril. With the resource timing too, you can also tell byte sizes, so then you can actually calculate bandwidth utilization like taking the total amount of bytes. Spreading it, assuming they’re loading uniformly so the rate of download is constant. You can project how much of the bandwidth is being used at any time interval. These are all guesses, but they’re fairly good and we actually have an example that we test this out.


  Something else with memory, in Chrome there is an API called window.performance.memory, if you go into right now type in window.performance.memory. It will give you a bunch of metrics telling you how much total RAM is available. How much RAM is being used right now. What is the maximum amount of RAM that can be allocated. These are good metrics, but they’re not absolute values, they’re like and estimate of what is available again Chrome doesn’t want to expose detailed values. It gives some idea, but not very close values. JS heap size, total JS heap size and used JS heap size. The used JS heap size is something that we care about in this particular measure that we’re looking at. If we plot the used JS heap size all the time. It tells us how many JavaScript objects we’re allocating and that gives us and estimate of how much memory our pages are using.


Nic: The JS heap size it’s reduced precision so it’s not an accurate count and it’s also delayed. These are both for privacy concerns. If you’re using this number it’s not more of an accurate number to use, but it’s something you can use over time to see if you’re leaking. You can use it between builds to see how the number changes to see if your application’s using more or less memory. The absolute number isn’t as useful because again it’s reduced precision, but you can just see how it changes and that’s an important thing to measure.


Philip: Something else you can look at is the total number of downloads of the byte size of the page. Just letting document get elements by tag name star, you’ll get a list of all the downloads and just looking at the length of that tells you how many downloads are on the page. Battery is another API that’s being standardized across browsers. Navigator.getbattery unlike all of the other APIs we’ve spoken about so far navigator.getbattery returns a promise. You need to call navigator.getbattery.then, except that’s not true on all browsers. Some browsers it just returns a battery object. By looking at battery value overtime you can tell if the browser’s charging or discharging … Yeah.


Philip: It might be, I’m not sure because I’ve seen it in various cases. Andy says it’s https only, it’s not too hard to verify since you’re all running without power right now. It will tell you what your current battery level is whether you’re charging or discharging and the time until empty or time until charged, I forgot what the timeline is. You can use that, trend it over time to tell whether you are using too much battery on certain operations like if you check before and after a certain function that call. Is that function using a lot of battery or not? You can also use it to determine if the users battery is dropping to a low, maybe I stop doing certain expensive things. If I was doing animation and now the battery has dropped below 20%, I’m going to stop doing that animation so that I use less battery. You want to talk about interactions?


Nic: Yeah, so the next thing in the time line of things to measure about the user after the page load is how are they interacting with the site? In the open this up and see there’s this interactions bar and basically it’s showing different scrolls, different clicks, different mouse moves, key downs, and stuff like that. That stuff is really easy to monitor on your own, there’s window document events for mouse moves and scrolls and all that kind of thing. These are things that you can measure, monitor and see what the user’s actually doing. There’s the visibility state which we talked about earlier, it’s good to know whether the user’s actually looking at your content or not. If there switching between tabs or it goes into the background, you can know that they’re no longer looking at your site.


Philip: There’s a new API available right now in the Chrome and Opera, the intersection observer that also tells you whether individual elements on the page are visible or not. You can typically use it to tell again if there’s an animation in your page or something else happening that’s not currently visible to the user, disable that. It can also be used to … I think advertising companies are very happy about this because they can tell if an ad is actually visible to the user or not.


Philip: They can tell if an ad is actually visible to the user or not by using the intersection observer. You can again use that to determine whether to pull in lazy loaded content or not based on what certain parts of the page are visible or not.


Nic: Similarly, there’s an API to know how a device is being held, so whether it’s in portrait mode or landscape mode. It’s more important, obviously, for a mobile; a tablet or a phone or something like that, but you can listening to the screen orientation or you can look at the screen orientation object, or you can also listen for different events. Maybe you want to relay out your page or something if the orientation changes from portrait to landscape. The other thing that is useful to measure over time are size metrics. Again, in the chromed out tools we have this chart over time and its measuring a bunch of different things. We’ll go over the things that it’s measuring here. One thing it’s monitoring is the overall size of the content that you have; the overall size of the HTML, how many downloads you have, how many images and scripts and all that kind of thing within your page. These all count towards the complexity of your site. How expensive it is to keep in memory, how big it is and how performant it could be, and especially if these change over time. Especially if they are increasing and not decreasing over time, it can give you a hint that maybe you have a leak in something; you’re not cleaning up where you should, and you’re doing something less performant then you could.


Philip: It’s also interesting to notice that your DOM doesn’t change, but your memory usage changes, you might have job script variables that are being leaked or something like that.


Nic: You can actually also, with the mutation observer that we have talked about, you can monitor how much of your content is changing over time. You can either look at the raw nodes but if the same amount of nodes are there and you’re actually just swapping in a lot of content, you’re actually doing a lot of work. You can use mutation observer as kind of a proxy for how busy your page is; how complex it is, by how much its getting fired every second or so. If you’re getting a lot of mutations, if you’re changing a lot of the content, you’re asking the browser to do a lot of different things, those are all things that you can monitor over time, improve on, that kind of thing.


Nic: And finally, its important to know about your site’s reliability. There’s an error function in the browser that you can look into and get notified of all the errors that are happening, and it’s good to know from a reliability perspective how many errors your page is generating in JavaScript errors.


Nic: We have a demo that basically tracks all these metrics that we just talked about. It’s a bookmarklet- Sorry, it’s actually a grease monkey extension. I actually don’t have it installed right here.


Philip: There is a link at the bottom if you want to try that out in your own browsers. You visit any page and it should give you that kind of a timeline so it’s measuring all those things that we just talked about. All the raw metrics we just talked about. It’s a live demo of measuring them over time, you know over throughout the page load and even after the page load as you’re scrolling and stuff like that.


  So what? This raw data is not really useful, right? We want to translate that into things like smoothness, responsiveness, and really, emotion. How does the user react? How do they feel about the experience that we are providing them? Are there ways to actually measure “window.get_user_emotional_state?” Anyone know about such and API? There is something called mind-reading markup language, that was published back in the late 90s. There are a few site that use it. It can read brain waves and stuff, but we haven’t had much success testing that out. Some things that we look at in terms of smoothness is how fast or how smooth are things like scrolls. We have a video here, right?


  In this case, one of the done by Bo Irish looks at before-and-after cases of in one case, there’s a lot of Jank, so you can measure that again by looking at Request Animation Frame to see how fast or slow things are moving, then looking at that after making changes, so in this case, I think they have to change- There was a big layout happening, and after removing that, the frame rate became smooth. Simon right here has a Jank detector that, again, runs through a site and measures how much- How smooth it is over time, so while you’re scrolling through the site, is it scrolling smoothly or not? Typically, if you have an infinite scrolling page, it needs to pull content in over the network. It might slow down while you’re scrolling just to pull that content in and then to render it. With this Jank detector- Do we have a URL to it? We have a URL at the end. Do you have a URL to it? He’ll Tweet it. Simon will Tweet the URL right now. When it detects how much Jank is on the page.


  Using things like detecting Jank or detecting smoothness, we can tell whether the user is experiencing something that’s delightful or frustrating to them. These kind of things are subjective, so just think to yourself, “Would I like something smooth, or would I like something Janky?” Then you can project that onto the user. Responsiveness is, how long does it take for the site to respond to a user action, so if the user clicked on something, how long does it take for something to happen as a result of that click? If they clicked on a menu item, how long did it take for that menu to display? If they tried to download something on the network, how long did it take for you to give feedback to them? In fact, how long did it take for them to get an indication that the click actually worked or not? You can use user timing to monitor your own code instrument; your single-page app.


  You want to give the user some kind of feedback within 100ms that tells them, at least, that we recognized what they did and we’re acting on it. We have a function here that- All it does is has a listener for a click, starts, runs performance to know when the click happens, and then runs a request animation frame to find out when the browser is ready to render something. All it’s calculating is the time between a click and the browser being ready to render. That gives you some indication of, how much time did it take for the user to think that that click was accepted. Just by saying the link changed color, or if you clicked on a button, how long did it take for that button to go down and come back up?


  Another thing you can look at is the E, so that the event object that’s past here, event handle, that has a timestamp on it; if you look at e.timestamp, it tells you when the browser actually registered the click versus when your click handler fired, so the delta between e.timestamp and performance and now tells you, how long did it take for the click to actually get to JavaScript? If it took a while, that means the browser was busy doing something else. Coming soon is something called a long-task API. It’s currently in development; not yet in the spec stage, so if you want to actually discuss this, you can and read comments on that URL, but essentially it’s, if any task took longer than- Was it 100ms or 50ms? I think it’s 100ms, then we’ll fire an API saying, “This particular task took more than 100ms,” and then maybe because it was queued up or maybe because it was a really long task, but it’s something that you can use later on to go and fix your tasks.


Nic: A task, in this case, could just be a JavaScript function being called, just like some timer being called, or some 3rd-party script executing something. Any time that that’s taking over the threshold of the few hundred milliseconds, you’ll get notified. It could be added to the proponents’ timeline that this thing on your site is potentially causing issues because it’s taking a long time to run.


Philip: Now we’ve collected all these metrics; can we actually map that onto a user’s emotional state? Can we actually simulate that “window.get_user_emotional_state?” It turns out we can make fairly good guesses. There are a bunch of research papers available, we’ve got links to all of them. First one is something called “rage clicking.” Has anyone ever experienced rage clicking? You click on something, nothing happens. You click again, nothing happens, and then you just start pounding the screen with clicks. It’s essentially a series of clicks in which users are pummeling their mouse buttons; it’s like punching you in the face repeatedly to get a reaction. It’s basically, they do that because what they expected to happen did not happen. Either they clicked on something that looked like a link, but was not actually a link, and therefor nothing happened, or they clicked on a link that didn’t work because some JavaScript failed or something else, so you want to measure these kinds of things to tell whether a user is frustrated with the experience or not.


  Very simple sample code; it’s not too long, it is small in size so that we could fit it on the slide, but again, if you have the slides, go on and look at the code. The bulk of it is just measuring whether-


Philip: You want to look at the code. The bulk of it is just measuring whether they clicked within a certain space repeatedly or not.


Nic: All right.


Philip: Dead clicks. Anyone experience dead clicks? When you click on something and nothing happens? Either because it required JavaScript and JavaScript hasn’t’ loaded yet or JavaScript failed. It was a JavaScript error and nothing happens because nobody looks at the web dev console when they’re actually using a site. You look at it after the fact if you are a developer.


  Has anyone gone and tried to fix JavaScript just so they could complete booking tickets or something? Your real users can’t do that right, but you need to know that if they could do that, they wanted to do it. This happened to Nic, I think while we were writing the doc. He tried to open his Facebook page and nothing would happen because the JavaScript required to fill in that comment text, comment field hadn’t loaded yet, so he clicked multiple times before it actually happened.


Nic: So the dead click here turned into a rage click because I was like, when is it going to let me do something? I just keep on hitting it over and over again. In this case it was Facebook on a mobile network and it just took too long to load the JavaScript to actually be able to react to me clicking. The page is loading, I type it in here. It looks like I should be able to click on the emotion. Nothing, nothing, nothing. Clicking, clicking. Nothing. It’s like oh, nooooooow. Nope, nope. Now, now! It took that long for me to be able to use the site in the way that I wanted to.


Philip: It’s interesting to compare these kinds of rage clicks or dead clicks based on the kind of network a user is on. Does it happen more often in mobile? Does it happen more often in other kinds of connections? If it’s consistent, it’s probably a development problem. If it happens specifically on mobile or certain ban width types then it’s probably related to downloading content.


  Then there’s mis-clicks. More often these happen on mobile devices where your click target is too small. So the user tried to tape on something but they missed it, so they clicked near it, but not exactly on the element and therefore nothing happened. How many times does this happen to you? It happens to me often. I have big fingers and a small phone. Unless you’re really good and grew up with a smart phone, you’re going to miss clicking on small text very often. Again, you can measure that. Just instrument the whole page and test if a click happened near a clickable element or not.


  Then there’s mouse movement. There’s also research that shows that people who are angry while using a site move their mouse in a very jerky and sudden fashion, but extremely slowly. So it’s jerky but slow overall whereas people who are frustrated, confused, or sad, are less precise in their movements and tend to vary their speed quite a lot. Just by measuring on mouse move over time, you can tell if the user is angry or frustrated or sad. You can’t necessarily tell if they were, that happened as a result of using your site, or maybe you can if you track it over time and you see on the first page they were fine and on the second page they were frustrated. Maybe you are the cause of that frustration, so fix something.


  Again, we have a link to the research paper if anyone’s interested in following up on that.


  Any questions about tracking emotion? There’s some things we didn’t cover like eyeball tracking and eyebrow tracking, which are also interesting, but we have links at the end if you’re interested in that. Specifically look at whether somebody’s eyebrows are furrowed while they’re using your site or not or does that change over time.


  Yeah, question?


Speaker 3: Have you considered the ethics of tracking people’s cursor movements?


Philip: Have we considered the ethics of tracking people’s cursor movements? We aren’t tracking them yet, and that’s one of the reasons we are not certain how ethical it is. What we would do is let a site decide if they wanted to and if they had the user’s permission to do that in fact. As a third party we wouldn’t do it automatically.


Philip: It possibly is, although the research is from a university NVU. Any questions so far about … Yep?


Speaker 4: How can you track when an animation frame is missed? How do you analyze that data or enter data tracking.


Philip: The question is, if an animation frame is missed how do you track and analyze that data to test if it’s meaningful or not. Typically a single animation frame missed may not mean much, but looking at it over time, if you see that you’re dropping a lot of frames or you have your frame rate dropping way below 60 frames per second. That gives you an indication that the browser is too busy doing things. If you expected an animation to run or a user is trying to scroll at the same time, they couldn’t’ scroll smoothly. If the user clicked on something and they expected a menu to show up. Taking that in conjunction with other parts of the page like where the dom change is required. You can tell whether a user’s action results in a smooth reaction or not and that is more useful than just looking at the frame rate on its own.


Nic: A lot of it’s about context. What were they doing before and you could attribute that to what happened afterward. If frame rate slows after the click, then it might be because of the click, you know, activity you did on the click. Related was that long task sepia that we talked about, which can give you an indication, can give you attribution for what scripts are taking longer than 50 or 100 milliseconds, and if you’re seeing those long tasks, events, happen at the same time scrolling is happening, you can actually see what script on the page is causing you to miss these animations. Again, that’s still in development, still being talked about, but hopefully that will give us that little bit more information to what can we do with this data once we collect it.


Philip: Some of our customers are interested in just looking at frame rate between first bite and onload just to know whether different scripts being downloaded are effecting frame rate or not. Are they suing up too much CPU? Is everything smooth? Should they be spreading out their script downloads over time or should they be concentrating them all at once. We talk a lot about paralyzing script downloads, but there are side effects to that as well which could effect frame rates, so those are good metrics to tell.


  How are we doing on time? We’re going to go very quickly on trusting your data. We’ve collected a lot of data over the years, and if you start doing this, you will end up with a lot of data. Question is how do you trust it? Is it valid data? Is it fake data? Is it just absurd? Are you in fact by your measurements effecting the overall measurement itself?


  The first thing we want to look at is avoid the observer effect. It’s a very common principle. If you try to measure something, you effect the thing you’re trying to measure. Heisenberg’s uncertainty principle comes in there. The thing with JavaScript is, it’s single threaded by domain. If you’re running JavaScript to measure things, you’re actually effecting all the other JavaScript that needs to run. Every time we inject a request animation frame or a set interval, we are effecting any other request animation frame or set intervals that had to run. How do we minimize the effect that we have or eliminate the effect that we have? By using things like navigation timing or performance start time that mark, we can get rid of the effect of measuring at that point of time before and until the browser is idle. There are other things we might like to measure when the onload event happened or when the dom complete event happened or when a click happened. The things that we want to do is do very cheap tasks in those particular handlers.


  If you’re listening to an event, just measure a time stamp, store it in a variable, and do nothing else. Defer everything else until the browser is idle.


  You also want to make sure the script you use to measure performance does not actually impact performance by slowing down load time. We mentioned this a little bit earlier, the Iframe loader technique where, what you do is you create an iframe, you wait for that Iframe’s onload event to file. The iframe is empty. It has absolutely nothing in it, which means its onload event files immediately. In the onload event of that iframe, you then go ahead and download your measurement script. That measurement script is now not effecting the load time of your primary page, but it is effecting ban width, so if you have a very large script, that is going to effect how much network is available for everything else you download. It is not going to effect the time it takes to download your current page.


  That moves your measurement code outside the critical part, so you avoid that. Another thing you can do is load your measurement code after the onload event so that you’re not effecting anything that has to load before onload. The problem with that is, you cannot then set a mutation observer before the page is completed loading. If you wanted to measure results timing for IOS our measure continuity metrics before onload event has happened.


  There are pros and cons to both what we’ve noticed …


Philip: There are pros and cons to both. What we’ve noticed is with the IFrame Loader Technique, adding the IFrame to the page cost about 80 milliseconds in overall load time. That’s across billions of page loads that we’ve looked at. 80 milliseconds is okay for most sites, but we do have a few people whose overall page load time is something like 700 milliseconds. 80 milliseconds is more than 10% of that. In those cases they do move our code after the onload event, in most other cases if you’re 3 or 4 seconds to load a page, 80 milliseconds doesn’t effect much.


  Like I said, do as little as possible in event handlers. Like set a time stamp store to variable, and then call a requestIdleCallback. That’s something we haven’t spoken about yet that I’ll talk about right now I guess.


  requestIdleCallback waits until the browser is idle that it’s doing nothing else. It has no frames to render, no other JavaScript to execute, no other events to process. If the CPU is idle, that’s when your requestIdleCallback is going to fire.


  If you save your time stamp or anything else you need from an event handler in a variable, you can then go and process that variable later on in your requestIdleCallback. Unfortunately this is only supported on Chrome and Opera at the moment, but there are a couple of shims that are available. There is a very complete shim from aFarkas, that’s the second link down there that uses set interval, set time up, but it also checks, uses a mutation observer to tell if the DOM is changing, in which case it defers your callback.


  Then, there’s a very minimal shim from Google. Why is it so hard to say that? That’s only about 10 lines long that does the bare minimum to make requestIdleCallback work. It’s not complying with the spec, but it works.


  Then, you also want to avoid a few things, the unload event is a little problematic, nothing can be differed from there, so you can actually call a requestIdleCallback from an unload event, because nothing will run after the unload event handler runs.


  You want to keep things simple. In this case you can actually save something to a variable, and do it later. What you could do is save to local storage or save it to a cookie, and then hope that the user is actually going to another page on the site so you can process that data on the next page. Or use something like a service worker to process the data. Do service workers affect the current page slow time if they’re running unload? I don’t think anyone’s tested that yet.


  Stay away from the scroll event. The thing with scroll event is it happens every time the user scrolls and it can affect that scroll performance. If you’re going to attach the scroll event, you want to throttle it, don’t fire in every instance of the on-scroll event, instead check to see at least 500 milliseconds have passed between calls to your event handler.


  You want to use something that I call de-bouncing or throttling. There’s a link at the end of our slides to what de-bouncing and throttling are all about. One of them fires the event first and then waits for sometime. The other one fires the event after waiting for some time.


  Whatever you do, never ever instrument clicks on in fact anything on a flash object. There is an interface between flash events and JavaScript events, and that is extremely slow. If you do this, you can easily add something like 5 to 10 seconds to your overall page interaction, so never ever do that. We learned this the hard way.


Nic: I think you should just never use flash.


Philip: Yeah, never use flash, but if you have to, never instrument it. Yeah, that’s the problem. Flash is slow, but you can’t measure it, because measuring it itself is going to be very slow.


  Finally we get to beaconing. I’ll let you take that.


Nic: Sure. We’ve talked about all these different data/


Philip: Any question so far before we get to beaconing?


Nic: All right. We’ve told you about all these different data that you can collect, right? Within the browser via JavaScript you’re getting navigation timing data, and resource timing data, and maybe frame rate over time and collecting errors and all this kind of stuff. It’s not useful unless you actually send it. Do something with it, right? You want to send it back to you. Maybe you want to send it back to your data warehouse, maybe you just want to put it into a csv file, something like that. Somehow you want to get this data that you’re collecting in the wild and your users machine and send it back to you. There’s a couple of ways of doing that, and they all have a couple of caveats, so it’s very important to understand what ways you can or can’t get the data back to you.


  We call this beaconing right? Packaging up data, putting it on something and sending it back to the server, that’s beacon. There’s different types of beacons that you can use. You can use an image form, xhr or the beacon API and they all have a couple of different caveats.


  An image beacon, all you’re simply doing is you’re creating an image dynamically on a site via JavaScript. Note your visitors will never see this image, you don’t actually put it in the DOM. It doesn’t actually need to be an image that gets returned. Basically you’re utilizing the image object, and you’re setting the source of the object to the URL that you want to send your data to, and you put your data in the query string.


  In this case, and I’m putting the page load time in the URL as a GET request. That’s why I’m sending the data from the customer’s machine back to my back end. The server can do whatever it wants with it, it can log it or do whatever, but this is the easiest way to send data back, right? It’s very easy, very lightweight. Browsers know how to make image requests. You’ve got 100% browser support no matter what. Like every browser in the world knows how to create an image. It’s a quick way to get data back to your servers.


  One thing is you could also return a 204 No Content response from the image on your server, and this just helps ensure that you’re doing the most performance way of returning data possible. You don’t actually send any content back, your server doesn’t even really need to respond.


  The only problem with this is that there’s a limitation with the size of the URL that you can use. You can’t actually post data to an image, right? There’s no payload that you can put on an image object to send a lot of data.


  The biggest problem is that older versions of IE 6 through 8 have a 2083 character limit for the URLs that you can use. If the data that you’re trying to send back to yourself is over 2000 characters or 2083 characters it just won’t work in these older versions of IE.


  Now, if you don’t care about IE that’s fine, but if you do have customers in general using different versions, different browsers, some of them maybe using IE 8 and if you have a lot of data that you’re trying to send to yourself, you can actually do it with an image request.


  Most other browsers … we tested up to 100 kilobytes worth of URL link. For some reason Android we noticed stopped around 81 kilobytes of data, but I mean that’s a lot of data to be able to send to yourself.


  One other thing to be kind of aware about that you can affect at all is that proxies in the middle might also have these character limits. Older proxies, especially like in a corporate environment or something like that, they may have these limits both in and you can’t test with that. The URL link itself you shouldn’t just put an unlimited amount of data on it.


  Also note, if you want to be able to send data back to yourself over 8 kilobytes, by default most web servers out there have a default limit of about 8 kilobyte URL. You can change it. These are configurations for Apache Nginx and JBoss to bring it up to 16 kilobytes worth of data on the URL, but if you own your own servers then you might have to make this change if you do want to send a lot of data back to yourself.


  If you can’t do an image beacon, what else can you do? If you have a lot of data, let’s say you’re collecting resource timing data that we saw earlier. You still compressed it, but it’s still 10, 15 kilobytes worth of data, right? Or you can create a form beacon.


  A form beacon is kind of somewhere to an image beacon where you’re creating something on the fly in JavaScript. You’re creating a form object in JavaScript. You don’t actually put it on the page, like the visitor never sees it, but it’s a way of creating a post. In a post-type of request lets you put additional payload. You’re no longer limited to a long URL, the URL can be really short, and you can put nearly unlimited data into the post, you could put megabytes worth of data into it.


  The code here is kind of complex and I would actually recommend not using post beacons. If you do need to use it, you can use this code that’s in boomerang source code, but it has a lot of problem. For example, one of them is that, some versions of IE can hang if you don’t get all the things right, specifically if it’s a … and on a server you have to return a 204 No Content, content length of zero, a content type and this X-XSS-Protection flag. If you don’t send all of that, IE can actually like hang the browser for up to 30 seconds. Very bad for your users experience, right?


  We’ve had this form beaconing in boomerang for a while and we’ve actually found a lot of incompatibilities with browsers when doing this, everything from the browser hang that we talked about, to beacons opening in new tabs, so like a new tab gets opened automatically.


Philip: It’s important to note that when you use a form post you need to post to a hidden IFrame. Like with an image you can just load an image in JavaScript and it will do nothing, but the form post you actually need to have a target that that post goes to, and so you …


Philip: … post you actually need to have a target that that post goes to. You need to do it with a hidden IFrame which sometimes shows up in the window.history. If the user clicks the back button and starts going back to the previous page, they go back to the previous beacon and that’s not always what you want.


Nic: Again, we have this code here, it does work in general but we just found so many incompatibilities over time that we’re probably not going to recommend anybody actually send data in this way.


  Okay, so ignore the form beacon. What else can you do? You can use an XHR, an XHR is another way of posting data to somewhere else. XHR, it’s very similar to the image beacon, you just create an XHR object, you can get or post the data once you package it up, send it back to yourself. It’s very easy, it’s very lightweight, just 3 lines there basically to send your data.


  You do need to update your server to make sure that you allow posting of data, especially if it’s on a different origin. You want to send the Access-Control-Allow-Origin tag, otherwise browsers will not actually send the data to a different website. You probably want to add the Timing-Allow-Origin tag that we talked about earlier, which lets you see the performance data for how long this beacon’s taking. Just return No Content basically from the XHR.


  The downside of this is that it doesn’t work in older versions of IE either. IE 6 and 7 don’t have any XHR, anything at all, IE 8 and 9, you can’t actually send a beacon to something other than the current domain. If you have a different server that you want to send it to, or for a third party analytics package that you want to send data to, you can’t actually use that. There is this XDomainRequest object that you can use in IE 8 and 9, but it has a lot of limitations and it doesn’t work very well. You can only send text request, you can’t send like URL form encoded requests and stuff like that. It’s kind of Janky to use.


  Also, you can only send a beacon to the same scheme as the host, you can’t send data from a HTTP site to a HTTPS site or vice versa, which is pretty annoying.


  There is all those ways of doing it. What other ways can we do it? There’s this new API that’s in Chrome and Firefox and hopefully going to be added to IE at some point soon. It’s called the Beacon API, and it’s basically built to solve a lot of the scenarios that we talked about and it also guarantees some reliability.


  Now, when you’re collecting data, when you’re collecting beacons, when do you send it out? Do you send it out right at the onload event or do you send it out later? If you’re collecting like frame rate over time, at what point do you send that data?


  People would often hook into the onload event, or the before onload event and send the data when the user is leaving, right? They want to collect all the data up to the point that the user is leaving, and send it out there. The problem is, the only way to guarantee the data goes out at that point is to synchronously block the browser from doing anything until your beacon goes out, and then returns. This is bad, really bad for performance, that user experience, if the beacon takes a second to go out, and another couple of seconds for the server to say, “okay”, the browser actually hangs in that amount of time, and that’s just not good.


  People who are doing these synchronous XHRs to get the data out, because they wanted the data no matter what, but they’re really affecting the user experience.


  This is what the Beacon API helps avoid. It gives a certain level of guarantee to the data that you want to send to yourself. You can queue up a beacon in the onload event or the page hide event or before onload event. Even if the user navigates away immediately really quickly, in the background, the browser is going to note that data and then try, even though the user is no longer on the page, and try to send the data back to your servers. It gives you a little better guarantee that you can get on a critical path of the user navigating away while still maintaining that the data is going to come to your servers eventually.


  The Beacon API can also be prioritized not to interfere with other traffic on the site or other critical resources, and on mobile for example it could be coalesced to make sure to not wake up the radio a lot just to send your analytics data.


  It’s a really good thing to use if it’s available in the browser. It’s a very incredibly simple API, in fact it’s just 1 line, navigator.sendBeacon, you pick the URL and you put your payload in there, and then all that data gets sent back to your server.


  Beacon API again is not on IOS or IE right now. You kind of need to do IF-THEN-ELSE statement to figure out what to do.


  This is what we recommend. If you want to send data back to yourself, if sendBeacon is available on the user agent, use it, because it’s very reliable, you can do all the data you want.


  If the total data that you’re sending is under 2000 bytes, just use the image beacon. Guaranteed to go out, like every single browser in the world uses it, or can do it. If it’s over 2000 bytes an XHR is available, so it’s over IE 10, use XHR to send the data.


  Then as a fall back you could use forms to send your data, but we maybe just don’t even care at that point, don’t want to worry about the incompatibilities.


Philip: The cool thing is that most of the browsers that don’t support XMLHttpRequest to send beacon also do not support navigation timing, resource timing, or any of the other APIs, so it’s likely they will have very little data to send anyway.


Nic: Then we talked a little bit about this, but when do you beacon, right? It depends on your use case. When do you want to send this data back to your server?


  In general, as soon as you have it available, send it out, that’s going to be the most reliable. The user could close their browser and you don’t have that data anymore if you didn’t send it out, right?


  For general analytics, if you’re just checking number of hits to your page or something like that, send the data out as soon as you get it.


  For performance analytics you may want to wait until the onload event or the SPA navigation, like immediately after the event that you’re measuring, send out the data.


  For this continuous metrics that we talked about, like frame rate or number of errors over time, or memory usage and that kind of thing, in general follow these 3 events: the page hide event, beforeunload event, and unload. Different browsers support each. The top one is the most reliable, the page hide event is the most reliable if it’s there to be able to send your data at that point.


  Basically, as soon as one of those happens, get your data out, because the user is about to close their session. They’re about to go away, or they’re about to navigate to another site or something.


Philip: Once you have got a data to a server, we still want to make sure that data is clean, right? We’ve tried our best to make sure we’re not affecting performance on the client, but we don’t know whether all the data we’ve received is clean.


  What we still need to do is a lot of data validation on the received data. Things like the data type and range of the incoming beacon data. One of the things we often see is somebody trying to XSS our beacon collector just by sending fake data or sending things that look like XSS attacks on the beacon. They’ll even hit the user agent string for example.


  Incorrect data, data that’s absurdly wrong. You can’t trust client timestamps, so your client timestamp might be wrong. You can only trust deltas between client timestamps, but don’t use an absolute timestamp to tell something on the server, because it could be like 30 days in the past. We’ve seen things that are more than 30 years in the past.


  Check for a reasonable rate of data input, so something like rate limiting. You expect your beacons to come in at the same rate that your pages are being viewed, or maybe a little more if you’re sending multiple beacons per page, but do a sanity check after a number of page views, of us is the number of beacons that you’re collecting to see that you’re not getting fake beacons from a third party that’s just trying to denial of service your system.


  Validate that your beacons aren’t being cross-site request forged. We’ve seen this as well. There are attempts where there are malicious and accidental CSRFed. The accidental cases are very amusing. We’ve seen cases where somebody liked the design of a site, so they copied the entire HTML into their own site, replaced the content, but left the beacon in there. They sent somebody else’s beacon from their server. That was technically a cross-site request forgery, but it was not intentional.


  There’s also cases where somebody views a site through Google Translate, or Google Search in cache, in which case the beacon comes from those servers, but it’s not actually measuring the user’s experience.


  Then there are legitimate people who are trying to attack your service, trying to inject fake data into your beacon collection system.


  You want to use something like an anti-CSRF token to detect … to make sure that the beacon is coming from somebody who legitimately visited your page and came from a specific page.


  Then, you also want to segregate known bot traffic from real-user traffic, because the patterns of bot users and real-users are different. Typically you’d be running your own bots like synthetic monitoring tests or your own QA test against your site, so you know which IPs they’re coming from, or which user agents they’re coming from.


  You’ll also have competitors who are running synthetic monitors against your site. You want to know all like: Keynote, Gomez, Webpagetest, they all have standard user agent strings or some headers in their requests that lets you identify those particular agents so you can separate out the 2.


  You could also separate out bots that you run versus bots that somebody else runs. There are also going to be badly behaved bots, so some of the bogus data we’ve seen is all time or values. Well all things that look like timestamps are changed to 9999999. We don’t know why or who does that.


  We’ve also seen timestamps which are more than 30 years in the past, so instead of being in the trillions, they’re in the hundreds of thousands of milliseconds from the epoch.


  We’ve seen requests that do not change over time, so if you look at a request from a particular user, it happens every 3 minutes, it’s always the same kind of request. That’s typically a bot, it may not be identifying itself as a bot, but you can tell based on its usage requests that are very regularly spaced.


  Sometimes they make exactly the same high value purchase. We’ve seen a case where a particular customer was QA testing their site, and so they ran a script that would do a purchase worth $36,000 every morning at about 3am, run the same purchase for about, I don’t know, the whole hour or something.


  Just by looking at that, it’s anomalous, nobody else is making that kind of purchase during that time, so those kinds of things should be filtered out.


  Sometimes we see page load time that’s negative, or more than one week long.


  Bandwidth that’s measured in terabits per second which at this point is a little absurd.


  We use some kind of statistical methods to identify these … We use the median absolute deviation or Tukey’s fences to detect if something is beyond a load time or a bandwidth or some other numeric metric is outside acceptable ranges.


  The median absolute deviation is kind of like the standard deviation, except, it’s way better for actually data that you’re looking at. Tukey’s measurement takes all of your data, looks at the interquartile range and decides whether something is too far outside that range or not.


  You can look at your past data to estimate what your current fences should be.


  You can also use something like triple exponential smoothing often called Holt-Winters smoothing, to determine whether your data is within acceptable range for the current period of time that you’re looking at.


  There’s something called Nelson Rules, which as Young said, all data is inherently random. If you get load time metrics, they should be random enough. If there’s not enough randomness in your data, something artificial is affecting that, and therefore you need to analyze that separately.


  We said don’t throw away your outliers, analyze them separately to determine if they are in fact outliers or bots or not.


  Finally, we’re not going to spend much time on this because we’re still researching this, but we also want to be able to measure things when there’s no network available. We can use things like service workers to inject our measurement code. When we intercept the request, and then measure things in cache, those result until the network is available to send it out.


  If you also have your own service worker, navigation timing provides worker start, you can see that in Chrome right now. Other browsers will catch up soon, when they actually add service worker support.


  You can tell when a service worker started versus when the other network activity happened, and so you can measure that.


  One thing we are not sure of yet is how do you distinguish a queued up request because the user was offline. Therefore whenever the beacon was sent, that anti-CSRF token that I spoke about, it has an expired timestamp. How do you distinguish that from somebody who is sending fake data with an intentionally expired anti-CSRF token?


  Those are things we haven’t figured out, and so we’re researching this, if it’s something that you might want to research as well if you’re trying to measure your own sites.


  There is the Web Performance Working Group. Anyone can join this. Nic knows the details about joining since he is a member of it.


Nic: Well, so the Working Group is in charge of a lot of the APIs that we talked about today, so navigation timing, resource timing, user timing. It’s really the community that’s helping build a lot of the stuff that we’re getting, and the working group helps solicit feedback from the community, from vendors, from people that are building websites to help understand what they want from both a monitoring point of view, or how to build better more performant websites. The Working Group is really the incubator for a lot of these specifications, and the reason that these specs are in all the browsers at the same time. All the browsers are working together to do it.


  If you’re interested in the stuff that we talked about, if you’re interested in giving your feedback and what you want, the Working Group, they have a mailing list and other events and stuff like that, and it’s really a great way to be a part of the community.


Philip: All right, finally we have a whole bunch of references in the slides, you probably already seen that, all the APIs that we covered and a whole bunch of links that have more information on the research topics we ran through.


  That’s all we have time for today. Thank you.