In this webinar, Gopal Brugalette of Nordstrom, and Buddy Brewer of SOASTA explain how the different segments of an organization—business, product management and technology, come together to prepare a site for peak traffic.
Hi everyone. Thank you for joining us. My name is Katie Stohlmann, I’m with O’Reilly Media, and I’ll be your host for today’s webcast. We’d like to begin our webcast today by saying thank you to our sponsor SOASTA, and let you all know in the hypercompetitive online marketplace, providing a better customer experience is the only way to win. It’s a world where milliseconds can add up to millions and one where performance is everything.
Optimizing customer experiences goes way beyond measuring speed and engagement. SOASTA gives you a crystal clear view of the relationship between end-to-end performance, user behavior, and business goals, and provides the actionable intelligence you need to maximize digital business performance in real time. SOASTA is a digital performance expert trusted by industry leading companies including Experian, Gilt Groupe, Hallmark, Intuit, Microsoft, and Netflix. With SOASTA, you spend less time searching for intelligence and more time putting it to work. Thanks again SOASTA.
Today’s webcast, Preparing Your Site for Holidays and Major Events, is being presented by Tammy Everts and Gopal Brugalette. Tammy has spent the last 2 decades obsessed with the many factors that go into creating the best possible user experience. As a senior researcher and evangelist at SOASTA, she researches the technical, business, and human aspects of web application performance, shares her finding via countless blog posts, presentations, case studies, articles, and reports. Gopal is a nuclear physicist, performance engineer, wood worker, farmer, and philosopher.
I’ll turn the program over to our speakers in just a moment but first, let me go over a few housekeeping things to help you get the most out of today’s webcast. First, you’ll want to open the group chat widget if you haven’t already done so. This is where we’ll interact with each other during the event and where you’ll submit your questions. We find that our audience usually has a lot of good knowledge to share, so we encourage you to chat freely during the event. However, if you have questions for our speakers, please press the symbol with the capital letter Q so we can make sure they see it when they are ready for Q&A. You can also open, move, and re-size any of the other widgets.
If you like to tweet from the Twitter widget, you need to give it permission to access your account. Our hashtag today is #VelocityConf, all one word. The Twitter widget will automatically append that to your tweets so you don’t have to. If you have any problems during the event, we encourage you to look at the help widget which is very thorough, but if you continue to have any problems, post in the chat room and one of our staff will help. For choppy audio or visuals, try refreshing your browser windows. Remember that the best thing you can do for the audio stream is close any apps that could be interfering. We are recording this event and we’ll send an email to everyone who registered when the recording is available, and that’s usually within 48 hours. Before I turn the program over to Tammy and Gopal, I’d like to introduce Jason Yee, our Velocity Community Manager. Hi Jason.
Hi Katie. Hey everyone. I wanted to let you know that registration is open for the Velocity New York and the Velocity Amsterdam conferences. Velocity in New York will be held at the New York Hilton Midtown, October 12 through 14, and Velocity Amsterdam will be held at the Amsterdam RAI, October 28 through 30. We’ve got a fantastic line-up of speakers for both conferences presenting on topics such as web performance, metrics, continuous delivery, micro-services. Tammy Everts, one of today’s presenters is actually going to be speaking at Velocity New York about metrics. If today’s webcast leaves you wanting to learn more about preparing and monitoring your site’s performance, I’d invite you to join us to Velocity. Thanks to today’s webcast sponsor, SOASTA, you can receive 25% off registration for both conferences by using the discount code SOASTA25. With that, I’d like to turn it over to Tammy and Gopal.
Thanks Jason. I’m going to pass the torch over to Gopal. Maybe introduce yourself a little more Gopal and then I will, and we can jump right into things. Just to let everyone know, we’re going to do Q&A at the end of the session. I think we’ve timed our slides so that we all have ample time for questions at the end. Please put those questions in the group chat and we will get to them all at the end. Thanks.
Hi everyone. Just a little bit about me. I spent about 15 years in Performance Engineering. Previous to Nordstrom, I was at IBM, IBM Global Services, and then after that, Washington Mutual. Previous to IT, I was involved in Nuclear Physics research. I worked in some projects in Japan, Europe, and United States at large accelerator facilities. Outside of work, I like to develop my permaculture farms. That’s me and my little tractor. I also have a small woodworking business making puzzles, toys, and other flying items.
That’s really cool. I like that dragon. I have been working in the user experience and performance space for the past 2 decades. The past 6 years, I’ve been focusing exclusively on front end performance. Before working at SOASTA, I was a performance evangelist at Radware, and before that, at Strangeloop Networks which was acquired by Radware a couple of years ago. I’ve not been a nuclear physicist. I don’t think I’ve done anything that kind of is in that sphere. My claims to fame in terms of … I mean, anything interesting to share with myself is that I live … The picture in the middle there is like the tiny town that I live in kind of the middle of nowhere in just west of the Canadian Rockies. If I do say so myself, I am an expert on all things to do with cupcakes, baking them, eating them. If you have any questions about any of those things, you can reach out to me on Twitter. That’s my handle up there, @tameverts.
I’m going to jump in to our first slide. I basically wanted to set things up and then kind of lob things over to Gopal who, I’m sort of the worker on the back and he’s the headliner. I wanted just to prime you for what he’s going to be talking about and just in general show stats around performance and their impact on user behavior and ultimately on business metrics. Then Gopal is going to go into the specifics of what that means at Nordstrom. Here’s like a laundry list of all the KPIs, the key performance indicators, that are affected by performance improvements or performance slowdown.
I’ve written pretty extensively over the past several years numerous case studies that shows that incontrovertibly there is a connection between performance and business metrics. I put a little question mark after every up there in the header, Performance affects every (?) business KPI, simply because if I’ve gone out of radical and tried to find a case study around performance in a particular metric, I always find one. I have yet to find a metric where performance wasn’t affected. If you know of any case studies in new direction where metrics weren’t affected or where metrics were, please let me know. I’m actually working on a book for O’Reilly on the business value performance, and I’m eager for as many case studies and real world examples as possible.
Just getting in a few, for example conversion, which is probably here an eCommerce shop or a SaaS vendor. This is probably something that you care about the most. This graph represents aggregate data, not Nordstrom data, the aggregate data that we looked at SOASTA, we gathered using mPulse, our real user monitoring solution. We aggregated this data and we found there’s really great connection, I love this graph, between the conversion rate and load times. I don’t know if you can make out the bottom axis, the horizontal axis is page load times and the vertical axis is conversion rate.
What we found, and I’ll actually go to my next slide, we pulled up the same data that is a little bit easier to see just looking at key points in terms of load times across that axis. You could see this really powerful connection between load time and conversion where at peak conversion rate, 2.4%, had an average load time of 2.5 seconds across the states we looked at. Then going down to 8.5 seconds where things kind of plateau in terms of conversion. I just have 1% conversion rate. If you’re new to just the whole idea of conversion to what that means, basically conversion rate is simply the percentage of sessions on the site of unique visits that end with the person converting. Going from being a browser to being somebody who buys something, downloads something, signs up for something. Whatever the call to action is for a particular page. This is a really compelling data here that is universal across the eCommerce sites. Here’s the data we gathered.
It’s not just revenue loss due to slowdown. We look at performance impact in 2 ways; we look at downtime and we look at slowdowns. Again, really interesting stuff from, this is findings from TRAC Research. What they found that the average revenue loss per hour downtime is $21,000, the average revenue loss due to performance slowdown, and they defined slowdown as the pages are rendering less than 4.4 seconds, or sorry, more than 4.4 seconds, was $4,100. While you would think that that means, “Okay, we should actually worry way more about downtime than slowdowns,” actually when you look at how much time on a site is actually, how much downtimes there actually is that accumulates for most sites, like maybe 1% or 2% at most versus performance slowdowns which can be just ongoing, it actually can come out even or even your numbers could be much worse on the slowdown side, but downtime is kind of sexier and gets all the headlines, so that’s what people pay attention to. My argument is that you should actually really pay attention to both.
The other metric that we see really being affected by performance issues is permanent abandonment rate. This is the data from Akamai where they found that if people came to your site and it was slow, there were 28% of the people would not return to the site, so your retention is just, it’s brutal like that, almost 1 out of 3 people just will not come back. Versus an outage, the permanent abandonment rate is only 9%. Again, if you’re thinking that you need to … You worry more about average than slow performance, again these are numbers that really invite you to think differently, things that you have to … You should care about both.
Just pulling up some headlines. I apologize if you work for any of these organizations and these headlines brings up any post-traumatic stress for you from the past few weeks. As you might you know, there was a recent state of outages and glitches amongst the major sites that really unfortunate reminders that outages and downtime and other glitches are inevitable. You can’t plan for some of them, but sort of like what kind of process do you need to have in place and what kinds of things can you do to mitigate, what kind of planning can you do in advance to prepare.
Of course, the most dramatic example of this of course would be the Amazon Prime Day Sale where there was a major outage that happened during a major shopping event. That’s basically what we’re going to be talking about or what Gopal is going to be talking about later in the session is the fact that now, more and more retailers don’t just have Black Friday and Cyber Monday as their main events and peak traffic times, but really there’s more and more retailers are having these events throughout the year. Really, you need to be in a state of readiness for all of these events. It’s not just about Black Friday anymore.
Obviously holidays are important and something that we’re all planning for and chances are … There’s a good chance that if you’re here at this talk today, Black Friday is what’s on your mind as you get ready to do that final push to prepare then lockdown all of your systems. Just looking at some numbers from comScore, you could see this dramatic surge. This is just desktop eCommerce sales. I didn’t put the mobile numbers in here. They’re also good numbers but the desktop numbers are really kind of what most people are focused on for now anyways, although mobile is definitely important. You can see this 50% increase year to year from 2013 to 2014, and based on these numbers, projecting that this holiday season, just eCommerce sales over the holidays at a 2-month window in November and December are going to probably exceed $60,000,000,000 which is really considerable and comes back to what I said earlier about the not being just one peak shopping event.
Again, these are comScore numbers where they looked at the numbers for 2014 and what were the top spending days in 2014, you can see Cyber Monday led the pack but actually the number of peak shopping days like Tuesday, which I think somebody came up with nickname for that after the fact, but basically the day after Cyber Monday was a huge shopping day. December 8th, Green Monday. Obviously, Black Friday. But then these other random spending dates, December 12th, December 9th, they’re just … There was a really great article that came out last year that just said, really the holidays are a series of peak events, so it’s not enough to be prepared for the one day. Obviously, to prepare for Cyber Monday, you’re probably prepared. If you can handle that onslaught, you’ll probably prepared for everything else, but it’s really not just about that one day anymore.
In terms of holiday order value, it’s how much people are putting in their carts and checking out with, these are the numbers from IBM. You can see this growth in cart size in 2013 and 2014, it’s about 19% growth, and extrapolating to 2015, we’re looking at something like … Oh, I can’t do Math on top of my head. Something like about $145. Really significant numbers there in terms of just how much people are spending online in one go. Extrapolating from that number, this Aberdeen study that found the impact of a 1 second delay. For every 1 second delay, a site experiences a 7% decrease in conversions, 16% decrease in customer satisfaction, and 11% decrease in page views. You can really see that making your pages faster versus suffering slower pages, you’re really leaving a lot of potential revenue on the table. On that note, I am going to pass the mic over to Gopal to take over from here and get into the specifics of how Nordstrom mitigates for all of this.
Thanks Tammy. I’ll go over just real briefly a little bit background on Nordstrom for those of you who aren’t familiar with us. We are a leading fashion retailer within the United States. We have a really big brick and mortar presence throughout the United States, Canada, and Puerto Rico. Then of course we have our major online site and our mobile offering which includes a mobile-optimized website and native apps on iPhone and Android. We have a very strong omni-channel presence. At Nordstrom our goal is deliver the best possible customer experience. Customer expectations though, they’re constantly changing, they’re constantly evolving over time. We want to let our customer be our guide and let them tell us what they expect.
Key to that customer experience is performance. By measuring, this is from our own data, we can clearly see what our customers expect in terms of performance from Nordstrom.com. Similar to what Tammy was showing, we know just from looking at Nordstrom.com that our peak conversion rates come in around 2 to 3 seconds. That’s what our customers expect. A lot of times people say, “Yeah, industry standards might tell us one thing but I want to know what it is for my customers.” Best thing to do is just measure and then you can very clearly see what your customers are expecting.
As Tammy showed, there’s plenty of data to show that performance is important all year long. It of course becomes a major focus at your major events. For Nordstrom we have really 2 peak periods. We have the holiday which is pretty much from Thanksgiving to about New Year. Then we also have the Anniversary Sale. The Anniversary Sale, I’ll just put a quick plug-in for Nordstrom. It’s going on now. You still have 4 days to take advantage. What the anniversary sale really is it’s when we have … It’s unlike a lot of sales that go on in the retail business which are just clearing out old merchandise. The Anniversary Sale is where we are actually discounting next season’s fashion, so fall fashion. It’s really great product or really great prices.
The Anniversary Sale, it lasts about 3 weeks total and actually for those 3 weeks, it disproportionately contributes to our revenue. It is a huge, huge, huge event for Nordstrom. It’s broken up into 2 parts. We have the main public sale, but prior to that we have what we call early access. That’s where members of our Nordstrom rewards program can actually access the sale prior to when it’s open to the public. A lot of our items sell out, and so, it’s a nice perk for our loyal customers.
How do we get ready for this sale and for these major events? We’re always getting ready. Anything that we ship into production, we’re already making sure that it will support our anniversary and our holiday loads. Anything that ships, we want to be ready, because as Tammy showed, those peaks, peaks can occur at times when you don’t expect them. Additionally, performance always matters. Not just concerning your peak events. Four years ago, this wasn’t the case. We didn’t have quite that focus on the customer experience. We were just really focused again on those peak events, making sure the site wouldn’t crash, not on individual customer experience and not fully aware of the impact that site’s page slowdowns can have on our revenue, on our customer experience.
Our customer experience suffered. We thought, with that focus on just the big events, we thought, “Well, anniversary sale is coming in a year, we won’t worry about performance. It’s coming in 9 months, we won’t worry about performance. It’s coming in 6 months, hey we know we have some performance problems but we have 6 months to solve them, 3 months, 2 months.” Then we had to go into that rush mode to get ready. Then being too that customer experience was suffering all year long.
What did we do to change that? We embraced a cultural shift. Three key points to that cultural shift. Number 1, performance, thinking about performance as a feature. We typically think about functionality as feature, but performance is a feature because functionality and performance are the key components, the key faces to the customer experience. When you think about performance on the same level as functionality, then you want to come to the conclusion or come to the consensus that everyone owns it. Performance is not then just the realm of performance engineers or ops. It’s the developers. It’s the product management. It’s the DBAs. It’s everyone on the team that really owns the customer experience. At Nordstrom, that’s our focus is always what is the impact to the customer. We realized that performance has that impact, and everyone needs to own it.
The second point being instrumentation and data. We need the APM tools. We brought in tools that can do the code profiling, that can do the log analysis, that can give us our own data so that we can actually understand what the performance is of our site. If there are problems, where the problems are, understanding what the root cause of those problems is. The tools that give us the data, so that we can effectively manage performance. Then as well as culture of continuous improvement. That’s about doing what we do, making it better a little bit at a time. Continuously or constantly getting better at what we do. There’s other processes that we brought in pretty standards, Kaizen’s value stream mapping, studying the ways, all different things that we can do to incrementally improve how we’re doing it. This results in a lot of more new automation, new tools, new processes, and it’s what really helps us evolve with our constant along with our customer expectation.
How do we get ready? Every feature at development, we are going through a process where we analyze it, we test it, and we monitor it. Again, we want to make sure that anything we ship is going to be able to support out peak load. When we’re doing this analysis, this is again where it’s really important to team across the organization, so really bringing together product management, everyone across the technology organization to ask, “What is the customer impact and what is the technical impact?” Sometimes the technical impact is easier we know will result in some more calls to the database, it’ll put some more load on these APIs, but really also understanding what is the customer impact going to be to the decisions that we’re making? How is it going to affect how they browse or flow through the site? This is actually probably the harder aspect and probably where we have had the most difficulties.
One example, a couple of years ago during our early access sale, we had a major issue and it actually ended up tracing back to decisions we had made months ago. Our security team has decided that we needed a stronger password criteria to better protect Nordstrom customers’ accounts and saw that the best way to do this was to force all these … They expired all the customer accounts on Nordstrom.com, so that would mean that the next time a customer logged in, they would be forced to reset their password. This is done many, many months in advance of the Anniversary Sale because we knew we don’t want to make too many changes during the Anniversary Sale. What we didn’t realize is that not a lot of customers would actually log in and change their password prior to the Anniversary Sale.
When early access came around, the key principle of early access is that the users have to authenticate in order to be qualified to browse the early access. Basically every user in Nordstrom.com now was channeled through the log-in flow, and because all their passwords had been expired, they all had to go through and do password reset. We now had all the users on Nordstrom.com going through a password reset and as luck would have it, the store procedure code in the database that handled the password reset was not optimized to handle such load, locked up, and prevented a very large percentage from our users of accessing the site. That was one of our more memorable experiences for early access.
People ask me sometimes, “What are you more worried about? Cyber Monday which obviously has really high load or anniversary?” I always say anniversary because in anniversary, we change how the site works, we introduce new features, we’re specific to early access or anniversary. We don’t see them live in production with real user traffic until the actual anniversary event. Starting at midnight when we turn on those features, that’s when all the new features get exercised and get a heavy load. It’s definitely more challenging.
One of the things you want to do when you’re analyzing is look at your previous event or look at your previous traffic. Here’s some relative load patterns that we see on Nordstrom.com for anniversary and for Cyber Monday. What we realized was these 2 events look very different. Definitely want to look at what your past information can tell you. If you’re doing your event yearly, it’s a good area, good source of insight, but realize that any event can … All the events can look different from each and events can change year to year. That’s why you want to certainly look at the path but understanding trending that’s occurring presently. A lot changes on the website, a lot changes for your business, a lot changes with users from year to year. Understand what new trends are so that you can try to model and predict what’s going to happen.
Even throughout an event, you want to understand what could potentially happen. Each event can look differently. Here’s an example of a conversion rate trend throughout the day. Here we can see that we have extremely high conversion rate at midnight. That’s when our really, really dedicated customers are logging in and they’re coming to shop and to buy. Then it starts to lower. Cyber Monday, we see that in the evening is when our customers start to really buy and we have really high conversion rate. Kind of the pattern of more browsing throughout the day probably comparing our competitor site. Maybe your customer, she’s at work, browsing through, thinking of putting different things in the wish list, into the bag, and then at night, probably after the kids are in bed then real shopping really takes off.
After all your analysis of what you think the event is going to look like, then you want to come certainly to the testing. You want to test in development, in perf, and in production. Test everywhere that you can to collect the most useful information at the right time. You want to test your projections and your contingency. I’ll talk a little bit more about that. The really key important point here is supporting your results. This is again another place where product management and technology really need to come together. Typically and again, you want to think about features, functionality, and performance in the same way. Very few people would think about shipping major functional defects to production but I’m always surprised at how many people will want to ship performance defects into production.
As Tammy really demonstrated with all that data that it’s potentially worse to ship performance issues. I always ask people … Shipping and performance … Or put it like this, shipping and performance defect will drive down your sales and your conversion. Will shipping that feature that has a performance defect, will that drive up the conversions in the sales greater than the poor performance will drive it down? Probably not. In some sense, you have a much greater risk of negatively impacting the customer experience than you do with possibly impacting the customer experience with a new feature. This is where again product management and technology really come together and understand the implications to the customer experience.
When you’re testing, it’s important to understand how to come up with your workload model. There’s a big tendency to go with worst case, but you really want to focus on the most realistic case. There’s a cost associated with having to support your worst case. We oftentimes start with, if every customer of the mobile app in Nordstrom logged on at the same time, what would that look like? We just know that doesn’t just happen and we don’t need to support every single person that’s ever downloaded the app, login into our site at the same time. If we did, we’d have to add a lot more hardware and do a lot more tuning in order to support. There’s definitely a cost to that. We really want to target what is the most realistic case. That’s where you want to … It comes back to the analysis that you’re going to do.
Then monitor. You can’t test for everything, you can’t analyze everything, but you can … Or let’s just say you can monitor a lot more than you can test and analyze. Critical too in this process is putting your monitoring plan in place early, make it part of your feature development. If we’re developing features, one of the key things we’re asking right at the design and coding is, how am I going to monitor the performance of this feature? Then we develop our monitoring instrumentation. We put it into our development and test environments, and so we’re actually testing and proving out our monitoring as part of proving out our code. Then when we ship to production, we ship the monitoring along with it. This is also another important area for the engineering team to team up about. You want to make sure that you’re understanding your bandwidth and making sure that the developers and the engineers do the monitoring because shipping features into production is not just about coding, it’s also about managing them and monitoring the in production.
Graph might be a little bit hard to read, but here this is an example of one of the monitoring dashboards that we use where for every release, we want to compare and look at our trends. You can see in the blue shading is what our current performance is after a release to production. Then the yellow line is we’re comparing that prior to our release. We’re looking at trends to make sure that our performance and our customer engagement are the same, and we haven’t negatively impacted them with shipping something to production.
After doing all those preparation, then of course the event will come. This is now where all your monitoring plans really come into play. You want to monitor not just system health but really the focus should be on the customer experience. That’s what matters most. You also want to do a contingency planning. Other times through testing, you might do what we call like breakpoint testing. You might know certain feature or certain API can only handle a certain load, but you anticipate that you won’t hit that load, but you never know what might happen. For example, we might turn off some of our personalization features if the load gets too high, so that we can control that load. Have those plans in place. Have those conversations with the product management, with the business before the events so that you have those plans if you do need to exercise them.
Then, you also want to coordinate with vendors. We had one example … Especially, this is … Most vendors are going to realize that during the holiday period, there’s going to be heavy loads and a lot of times they will go into lockdowns. For your special events like an anniversary sale or a particular sale event you’re having or marketing event, your vendors might not aware that you’re doing this and they might just plan to do a change. Sometimes changes are impactful, so that could really affect your site. An example of this that we saw is that when you’re doing anniversary, we saw that our order management system was starting queuing up all our orders. It wasn’t processing orders anymore. Initially we thought, “Oh, there’s some sort of performance problem,” but when we dug down, what we found out that the vendor we used for our projects to make sure our orders weren’t fraudulent had implemented a change just a few hours before anniversary event kicked off and that change did not go well and it was actually blocking all of our orders from going through for anniversary. Now, we do a much better job in special events and making sure that they’re ready as well.
This is where monitoring is really important because what we’ve seen is that the issues happen where we least anticipate them. In this recent anniversary, we had again early access where early access only for basically one week a year, so it’s very hard to predict what’s going to happen. What we saw was that we had almost 10 times more mobile sign-ins than we anticipated. What ended up happening, so this graph is really interesting, it kind of show us that we’re having around 4 to 5 times more than we anticipated and then it goes up to almost to 10 times. During this time, we were able to handle the load but we ended up having … This unexpected traffic triggered some of our security framework and so we ended up blocking a large percentage of our users who are trying to log in and they actually received like a access denied error.
It took us a while to figure out what the root cause of it was, and then once we fixed it, then we saw that what the actual demand was actually pretty high. Again, issues are going to happen when you don’t anticipate them. Sometimes people ask me, “Are you ready for the event?”, and I say, “I am 100% confident that everything we tested will work just fine,” but if we didn’t test it, it’s hard to know how it all works. Monitor and see where your … When loads are different than what you modeled, that’s probably where you’re going to have problems.
Again, in summary of how we prepare our site, Nordstrom.com, for our major events, first is really that cultural shift where now we take performance as a feature. We implemented all the APM and performance management tools that we needed to give us the data and we implemented a continuous improvement culture. We get a little bit better at what we do everyday. Then we’re always getting ready when it comes to performance. Every feature that’s going to ship, we know it has to support our peak loads and so we’re always preparing. We go through a process of analyzing the feature for customer impact and technical impact. We test and then we monitor. Looking ahead to the future, we’ll always have the continuous customer focus. We’re moving more and more towards continuous delivery. We’re looking at implementing more cloud-based architecture, and of course, our continuous improvement will continue. That’s our … I guess, did we want to take questions now?
Yeah. Maybe I’ll mention this and then we can do some Q&A.
If you’ve missed the beginning of this talk, just a little bit about SOASTA. We are a provider of performance testing, monitoring, and analytics solutions. We service the top 20 retailers, so companies like Target, Walmart, Best Buy, and obviously of course Nordstrom. What we would like to offer at SOASTA for everyone who has attended this talk is a free holiday performance readiness assessment that includes baseline performance test using our SOASTA CloudTest, a site performance analysis with one of our performance consultants who will help implement the baseline performance test, we’ll also be giving your complimentary mPulse real user monitoring through the rest of 2015, so getting you through the holiday session, and a summary report with test results and optimization suggestions. If you follow that URL, SOASTA.com/holiday-readiness, that’ll take you through the link to sign-up. It’s pretty easy. If I do say so myself, it’s a really great offer so really, if you’re interested in analyzing, testing, monitoring, that trifecta of skills that Gopal was talking about, then you should consider checking this out.
Looking at the questions that have come up so far. Sarah Stevens asked, “Have you been able to get more specific performance information around page groups rather than performance statistics? For example, is it more impactful if the entry page, home, or shop page is slow versus the cart?” That’s a real great question. I’m so glad you asked. Gopal, if you don’t mind, I’ll answer and then if there’s anything you want to add that’s specific to your experience, that would be great. I’d actually been writing about this for the past sort of while, so I’ve written a couple of recent blog post. I will throw the links into the chat window for you to check out. There’s one. Here is the other.
Basically short answer, yes. Conversions are affected differently depending on what type of site it is, what page is experiencing the slowdown. The first link that I shared is for a post that I wrote for our blog, The Performance Beacon. PerformanceBeacon.com actually is our short link. It’s called, “When it comes to delivering the best possible user experience, how fast is fast enough?” What we found, again looking at data from different eCommerce customers of ours is that if you have a specialty goods site versus a general merchandise site, people are more willing to be patient if your site sell specialty goods than if you’re a general merchandiser.
That just sort of intuitively makes sense if you think about it. If you’re kind of the only shop that sells that you sell, then obviously people are going to know that they have to be patient, but if you have a lot of competition, people are much more likely to bounce. We’ve had some really interesting numbers there. We ran both bounce rate and conversion. You can see the charts in that blog post.
We also found that people are more patient at some point in the sales funnel than they are at other points. For example, when people are kind of in the browse phase of the transaction, so that could be landing page, product page, category pages, if load time drops from say 1 second to 6 seconds, conversion rate drops by 60% which is hugely significant. But then if we look at the same set of user data, we saw that the impact on conversion was really reduced if it was the check-out page that degraded speed.
I think it’s interesting, just looking at the group chat window, we see with Jeremy, I’m sorry, sorry if I’m mispronouncing your name, you see that for you a cart delay would be more impactful as you have many entry points you say, but all orders go through the cart. There could be exceptions and this is really where it comes down to looking at your own real user data and why we recommended that every site should be collecting real user data, if not through our solutions then at least through somebody’s solution, because it’s only through looking at your data that you will actually get a true sense of what people expectations and behaviors are.
Sort of related, we found that actually also visitors in some countries are more patients than others. Bounce rate and conversions are affected more for example by slow performance is somebody is browsing your site from the US or people have higher expectations extensively versus if they’re browsing from Australia where expectations are much lower. It’s interesting stuff there.
Sort of related to that is the second link that I sent which is about conversion impact scoring and kind of getting into. This is something you can do with any kind of real user monitoring tool where it use the algorithm that calculates which pages are affected more by performance than others. It might not necessarily be your slower page, it might be pages that are actually fairly performing but are still much more sensitive to changes in load times. The post kind of talks through an example of that study where you can see why if you … In an ideal world, you would like to optimize all your pages and have your page load in under a second, but actually conversion impact scoring lets you see which pages are the ones that you should really focus on, the pages with the highest conversion impact score obviously are the ones you should look at, and they might not be your slower pages. Hopefully that helped answer that question. Gopal, do you have anything to add to it?
I’m going to just add that certainly on our site, the lower you go on the funnel, the longer our response times are. I think that there’s certainly an expectation, at least for some users, that going down, the site will slow, the site performance will be a little bit less. The other interesting thing too is that we talked about the funnel is, the funnel doesn’t always look like what you’d expect. We have a very large percentage of our customers that don’t come in through the home page. They’re getting referred from searches or from other pages or from our online advertising. Really understanding, as you’re saying Tammy, what is the most impactful pages. Really understanding how traffic, how your customers are coming in to your site is really important so you know where to focus your performance efforts on because it may not be your home page.
Does anyone have any questions further to that? Okay. I guess I’m wondering if anyone has any specific questions or general questions really about planning for the holidays or even sharing what are the major events for your organization. Gopal and I are definitely not the only experts in the room here, so if anyone has anything that they would like to share in the group chat, we can talk about that. I’m always interested in knowing, as Gopal shared with his anniversary event, what are the significant events outside of Black Friday that affect different organizations and how you deal with those. You can be anonymous if you want. You don’t have to name your company. Well, it doesn’t look like there are any more questions right now. Any parting words that you want to share Gopal. If you could summarize all your wisdom into a parting shot.
I guess I would just go back, let me see here, back to this slide. Develop that culture around performance and then always prepare. I think one of the lessons we learned was the major events are really important but as we’ve seen, there’s all the data to show that performance matters every time. That every times people will talk about delighting the customer, and so, definitely performance is a key component of that customer experience and creating that customer delight that a lot of times is a goal for us.
I would agree. One second, there’s another question coming in. Kenda Sumi asks, “Tammy, how do we handle in the case that we don’t know the exact expected traffic for the event? How do you successfully model the performance test scenario?” I know that’s more a question for you Gopal. How would you do that?
You really want to start from what information do you know? A lot of times, we have a really hard time projecting what mobile traffic is going to be like or for example, we did a launch and integration with Pinterest where now users of Pinterest can click on one button in Pinterest to purchase Nordstrom items if they’ve been pinned by someone. This is like something entirely new. We’ve never done a Pinterest integration. We had no idea what those volumes are going to be like.
We tried to relate and I did some of this work. I tried to relate it to information that we did know. We did know what are conversion could be based on … I showed some graphs. We have a pretty good idea of what our conversion rates look like through different channels. We modeled, “Okay, what if?” Then we kind of knew what current traffic through Pinterest was. We did a lot of sort of what if models like assuming, starting that sort of a worse case. I talked about worst case. We started at the worst case where we said, “What if every person who went to Pinterest actually bought …” It’s funny we use the term worst case. Maybe this is the best case. Every person who went to Pinterest and clicked on a Nordstrom item and actually bought it versus in sort of a high water mark, and then comparing that to what’s probably a realistic case of what our current conversion rates are.
Then we did a series of what ifs. What if it’s 10 times worse? What if it’s 30 times worse or higher? I should say higher, not really worse. What if it’s 30 times higher? What does that look like? Does that kind of give you a sense?
It’s really about trying to relate to information you do have and then doing different scenarios. A lot of times in performance testing, people will do what’s called like a stress test or a breakpoint test where you want to see exactly what your system can handle. A lot of times, you can put in controls so that if you go above that, you can protect your site in different ways. There’s throttling. There’s a whole bunch of different tools you can use or approaches you can use. Then as well, I talked a little bit about contingency planning. Are there features on your site that are more performance intensive? Can you disable those features during your peak events to protect your site and make sure you have a good experience?
It’s interesting too, and that can actually be a … It’s a combination of technology as well as business. We have one feature we call BOPIS which is buy online pickup in store. In previous years, that was pretty intensive, resource intensive for our systems before we optimized it. We would actually just turn it off during our peak events. That was one way we would handle that. When you don’t know the exact expected traffic you really just want to start with what you know and then look at different models and see what is reasonable, and then do testing to determine what you can support. If you can only support 1 1/2 times your most realistic case, you might be more concerned, but if you can support 10 times then you’d have less concern.
Sounds good. I think we are exactly at 11:00 now. If there are no new questions right now, then I guess we’ll sign-off. I just want to remind people that if you do think of any questions after the fact, happy to take your conversation over to Twitter. You can find me @tameverts, T-A-M-E-V-E-R-T-S. I will be writing a follow-up post about this talk, which will go up on my blog performancebeacon.com. It’ll be going up next week, so I encourage you to check that out. Here I am right here. You can find me there. Thanks Katie. Katie posted it in the chat. I guess this is from us. Thanks very much everyone for your time and for taking the time today.
Thank you so much Tammy and Gopal for presenting such a outstanding webcast for our audience. Thanks everyone for submitting questions and joining in the conversation. As we close out, we’d like to say a big thank you once again to our sponsor SOASTA and let you know that in the hypercompetitive online marketplace, providing a better customer experience is the only way to win. It’s a world where milliseconds can add up to million and one where performance is everything.
Optimizing customer experience goes way beyond measuring speed and engagement. SOASTA gives you a crystal clear view of the relationship between end-to-end performance, user behavior, and business goals, and provides the actionable intelligence you need to maximize digital business performance in real time. SOASTA is the digital performance expert trusted by industry-leading companies including Experian, Gilt Groupe, Hallmark, Intuit, Microsoft, and Netflix. With SOASTA, you spend less time searching for intelligence and more time putting it to work. Thanks again for joining us everyone. This will conclude our webcast today. Goodbye everyone.