KentuckyDerby.com Crosses the Finish Line

CloudTest® Ensures Reliable Website Performance on Derby Day

Often called “The Most Exciting Two Minutes in Sports”, the Kentucky Derby is a Grade I stakes race for three year-old Thoroughbred horses, held annually in Louisville, on the first Saturday in May. The event caps the two-week-long Kentucky Derby Festival and is widely considered the most prestigious horse race in the world. Alternatively referred to as “The Run for the Roses” the Kentucky Derby draws an average of 150,000 visitors each year, including residents, out-of-towners, celebrities, presidents, and even members of royal families. On race day they ranked as the most popular Google search term, outranking the next most popular search term by a factor of 3900%. How did one of the most popular sporting events in the world prepare its website to handle hundreds of thousands if not millions of visitors?

“Having the confidence that our website will perform is essential at CDI, and the only way we know how to do this is to performance test with CloudTest” – Eric Eatherly, Senior Director of Web Properties, Churchill Downs, Inc.

Despite the economic recession and very challenging conditions in the U.S. Thoroughbred industry where on-track wagering of live racing is relatively flat, Internet betting, viewing and participation in the Derby has increased dramatically. In fact, last year during the hour leading up to the Kentucky Derby and immediately following, kentuckyderby.com received an unprecedented number of users. This large spike in traffic over such a short a period of time prompted Churchill Downs, Inc. (CDI) to test the performance of the Derby website in advance of Derby Day. Specifically, they wanted to understand the performance and scalability of their current in-house infrastructure, gauge any performance improvements with the addition of Akamai, and achieve approximately 10-12,000 HTTP hits/second while maintaining optimal response times.

The CDI technology team subscribed to a regular internal performance testing practice using open source tools and two web servers. Realizing that internal testing can only replicate a limited amount of load, CDI turned to SOASTA CloudTest to test outside their firewall and better emulate the anticipated large-scale volume. Time was of the essence as CDI wanted to have the site tested and ready before “code freeze”-one week prior to race day.

Understanding the Customer Experience

The SOASTA performance engineering team worked with the Senior Director of CDI Web Properties to determine the most trafficked areas of the website and quickly create the user scenarios to be tested. Because the Kentucky Derby is a national and worldwide event, user traffic is very dispersed, and with Akamai in place this became a key point in planning out the load testing strategy. To ensure traffic to the site was properly dispersed for testing the production site the load was provisioned as follows:

• 4 locations
• 9 zones
• 2 cloud providers

It was found that the average visitor hits six pages, after initially browsing the home page, and usually clicked on the “Contenders” section of the site to get background information on the horses and their race history, jockeys and owners. Visitors also frequently click on the ”Celebrities” section and browse through a photo series that showcases famous celebrities, royalty and presidents who have attended the Derby. Another scenario was created to test the “Derby Nation” section of the site, an online communications channel for subscribed visitors to catch all of the breaking news, “Derby Dish” and updates on year-round entertainment at Churchill Downs Racetrack. Finally, SOASTA created a scenario for the “Live Odds” section of the site where visitors go to get up-to-the-minute betting odds.

Pinpointing Performance Problems

Performance tests were conducted using the above scenarios which included log-in to the overall website and Derby Nation channel, the ability to join the Nation, as well as comment on a news item and browse through the “Party” section of the site that visitors access approximately every two minutes on average. KentuckyDerby.com typically sees 800,000 page views per hour (on non-race days).

The SOASTA performance engineering team conducted a series of performance tests using the defined user scenarios while increasing the traffic to understand the performance and scalability of the application and infrastructure. The ultimate goal was to successfully serve 25,000 concurrent users. As issues were found, while the test was running, the CDI team made the necessary changes and testing resumed with increased load. The team initially tested the site with 10,000 users and noticed that their two internal web servers were oversaturated and causing errors. To achieve better throughput, a third, larger web server was added to the mix and the load balancers were tuned to properly handle the traffic across all web servers, which resulted in better performance at 10,000 users.

KentuckyDerby.com a Lock on Race Day

To ensure the final Derby Day configuration was properly configured and tuned, Akamai was turned on for the final 10,000 and 25,000 user load tests. The testing with Akamai identified further areas of the site that required changes and updates, which the CDI team made during the performance test. After multiple rounds of testing, infrastructure tuning, and configuration changes, the final Derby Day environment was able to handle over ten times the throughput as compared to the first test run. The final 25,000-user load test experienced sub-second average response times and achieved the goal of almost 12,000 HTTP hits/second with over 100 gigabytes of data transferred in an hour.

Selecting the right load-testing tool to meet these varied scenarios and provide accurate results was very important to CDI, particularly given the near real-time aspect of the Derby. Their goal was to ensure that kentuckyderby.com served up both static and dynamic content, video footage and race day information to deliver an experience on par with and at the quality and speed of the live and televised event.

“Keeping our site up and running during a live event where instant load spikes can occur is a testament to the need for rigorous website load testing,“ said Eric Eatherly, Senior Director of Web Properties, Churchill Downs, Inc. ”As such, it is critical for us to know at all times just what our site will do and what our online users are experiencing, particularly in the time leading up to and immediately following the race. Having the confidence that our website will perform is essential at CDI, and the only way we know how to do this is to performance test with CloudTest.”

CDI chose SOASTA because its CloudTest on-demand service realistically simulates a variety of user behavior and traffic, accurately monitoring and measuring the entire web application and infrastructure including content and services delivered by third party vendors. CloudTest Analytics is based upon the first real-time Business Intelligence engine purpose-built for Performance Management.

Lessons Learned

• Event-driven sites experience spikes in traffic-some anticipated and some not. CDI knew it was important to understand the breaking points, define the capacity thresholds, and know how to react when those thresholds are exceeded. Testing the production site with actual anticipated load levels is the only way to understand how things will behave when peak traffic arrives.
• Test with and without third-party integration. Testing without third-party systems allows for isolated system testing and tuning, which was key for CDI. Otherwise, the need for an additional web server and load balancer configuration changes would not have been identified.