Tag Archive for ‘amazon ec2’ rss

Finding Bottlenecks Through Monitoring and Iterative Load Tests

Posted by Yuriy Verkhovoy on Jan 05, 2012. Leave a Comment

A medium-size web development shop engaged us for its first experience with external, scale testing on a brand new application scheduled to roll out a few months after we started. The application is written in Ruby on Rails (ROR) with a MySQL backend, and serves as an internal social networking hub for a company. Through this application, users can create projects and tasks that are both work related and personal, invite others to join them in completing these tasks, share their calendars, and exchange instant messages with other online members. The front end is entirely JavaScript driven and consists of a number of widgets, all of which are updated through AJAX calls to the server. In addition, the client intends to host this application in the cloud, which would be a first for them.

The test consisted of 10 scenarios including simple browsing of the site, creating projects and tasks, and randomly sending instant messages to other online members. The most challenging part of test creation was dealing with the asynchronous nature of the application. There were no clear-cut events that broke the script into URLs or pages. There was only one URL followed by a number of AJAX calls to the server. CloudTest’s nested transaction grouping allowed us to put these requests into logical groups that represented all of the main business processes to be tested.

The first test was against a single server that contained all of the moving parts of the application and was hosted onsite. The application performed well until we reached 200 simultaneous virtual users. At that point, things got really slow. At about 300 users, the site was unusable and every request came back with a HTTP 500 error.

Our second test was against a more distributed environment. The application server and the database were split between two different machines, which were hosted in Amazon EC2. The site did slightly better, topping out at 300 users. Using CloudTest’s monitoring, we noticed that the database was not working very hard and that the web server was at nearly 100% CPU utilization for most of the test.

During subsequent tests we tried a variety of configurations. CloudTest makes it very easy to do iterative load testing. We spread the load even more by putting several application servers behind an EC2 load balancer and increasing the horsepower of the database server. Although most of these changes made a visible improvement, we were still faced with an obvious bottleneck. At roughly 800 simultaneous users, the application started to slow down, and at 1200 users it became almost unusable. At this point all of the usual suspects looked innocent. The app servers’ CPU never got higher than 50%, the database had plenty of breathing room and we were not limited by bandwidth.

When we looked at the “Collection Summary” report from CloudTest we realized that one particular transaction, which was taking longer to complete than others, needed more attention during the test. As you can see from the chart below, the “CREATE TASK” transactions took significantly more time to complete, even when there was little load on the system.

The ops team monitored the app server with New Relic, which has a ROR profiling module and is integrated into the SOASTA dashboards. We found that a CREATE TASK business process was using a ROR ORM Engine called Active Record to perform MySQL queries. A particular query in the CREATE TASK process was not “tuned”, and instead of running one or two SQL statements to retrieve the data, hundreds of queries were run each time a list of tasks was requested by the application. Although the queries themselves were very lightweight, running hundreds of them in sequence for each process slowed things down considerably.

Our findings were brought to the attention of the development team. The team was able to revise the process to access the same data with just one query to the database. As a result, in our subsequent tests we saw a great improvement in the CREATE TASK transaction. Monitoring the infrastructure during a load test once again proved invaluable, and having the results on the same timeline helped surface the issues much more quickly.

 

Traditional Vendors Out, New Players Emerge in Cloud Testing

Posted by Tom Lounibos on Jan 30, 2009. Leave a Comment

Every recession over the past 30 years has proven to be a forcing function for significant technological change.  In good times, corporations can get a bit careless or lose focus. However, in bad times, corporations go to great lengths to find new levels of cost efficiency in their operations . . . even if it means changing processes and technology that they had used for many years.  This recession is no different, change is beginning to happen, and traditional software vendors are the prime targets. The traditional “License” model employed by traditional software vendors is officially dead with this recession. The “Software as a Service” business model or even more specifically, the “Pay for Use” model, have emerged as clear winners.  IDC recently raised its 2009 projections for the SaaS market. They now expect this segment to grow by more than 40% this year.  This business model shift, coupled with the emergence of the new deployment platform of Cloud Computing will make it very difficult for traditional vendors to maintain their thrones.

Take my business, the test marketplace.  Our market was long dominated by Mercury Interactive which was bought by HP several years ago.   Mercury’s LoadRunner was the ideal tool for testing client-server applications in the late 80′s and 90′s, and its success drove Mercury to its leadership position under a license-based model.  However, a license model that was very expensive ($30k/Test Hour). Today, the world has changed for HP/Mercury. We are all now developing and deploying new and much more dynamic Web applications for consumers around the world. Mercury’s technology, business, and deployment model are looking a bit like Tom Jones performing at a Beyonce concert . . . a little out of place. New test vendors are beginning to emerge as the new leaders of the Cloud Testing market.  This includes new players like SOASTA that deliver a “pay only for test time used” ($1k/Test Hour).  Companies like these are greatly reducing the cost of testing while even enabling more and better quality testing. This is an example of another changing of the guard, made possible by another down economic period and a requirement for greater reliability.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Genentech Tests Cloud Computing

Posted by Tom Lounibos on Jan 29, 2009. Leave a Comment

As Michael Liedtke of Associated Press reported last week, and subsequently now we can also report, that indeed Genentech has migrated its 16,000 employee’s to Google Apps for its email and for some office suite applications. However, the more interesting back story to this article was before they made the decision, Genentech wanted to ensure that Google would scale. Google . . . scale?  That’s right they questioned what kind of user experience there employees would have when accessing applications deployed in a cloud. To get comfortable, Genentech decided to do some Cloud Testing using SOASTA’s new On Demand CloudTest Service. Over the course of a couple weeks Genentech were able to test several different user scenarios including having 16,000 simulated Genentech employees hitting Google Calendar at the same instant.  The testing went well, and the rest is history. Genentech has dipped their big toe into the Cloud Landscape, and by all the feedback we have received,  it went very well.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Cloud Computing is Changing How we Test Web Sites Forever!

Posted by Tom Lounibos on Jan 28, 2009. Leave a Comment

Okay, I know I’m biased, but as the rest of the blogging world focuses their attention on the “Cloud Platform Wars”, debating endlessly on which Cloud platform vendor will emerge as the “winner” in the coming years, quietly, several companies (from young start-ups to enterprise class) are leveraging the access, availability and affordability of Cloud Computing for testing their Web sites.  Even more amazingly, they are testing without even considering which Cloud platform they are using.  They are Cloud Testing!

Load, Performance, and Stress testing Web sites has been a black art of Web development shops for years. . . requiring a huge amount of compute power, expertise, and expensive system and test software. The total cost of Performance testing has reached an estimated $30,000/test hour. . . making testing costs prohibitive for most companies. That is, until Cloud Testing emerged in 2008.

Now companies are leveraging the limitless power of Cloud Computing to simulate Web (scale) traffic to test their sites before going into LIVE production. They are greatly reducing the occurrences of performance related errors, latency, and actual site crashes while reducing the cost of testing down to $1,000/test hour.

So, while the rest of the world tries to figure out whether or not Cloud Computing is for real, we are seeing many customers seeking a higher level of reliability for their web sites,  at a lower cost, finding a real value proposition in Cloud Computing.

Cloud Testing is Changing How we Test Web Sites Forever!

contact me at: tlounibos@soasta.com; twitter.com/lounibos

SOASTA Performs 500th Cloud Test

Posted by Tom Lounibos on Jan 27, 2009. Leave a Comment

It seems like ten years have past since our first Cloud Test, but last night we hit the 500th test milestone with a test for Hallmark.com.   An amazing stress test that lasted a little over an hour and simulated over 600,000 users hitting their site.  The analytics (1 TB) were amazing in their detail of how the site and network would perform in live production.   Looking back at our first test,  we were so inexperienced in this new method of testing.  We had no idea what Cloud Computing would bring to Web testing.   It has been an amazing year of discovery and experience.  In the end, we once again re-discovered the point of testing is not in the ability to run a test….but the real point of testing is in the Results and Analytics that the test produces. Cloud Testing is changing testing forever. Thank you to all of our very loyal customers!

contact me at: tlounibos@soasta.com; twitter.com/lounibos

If Cloud Computing is The New Destination, Then Where does the Journey Begin?

Posted by Tom Lounibos on Jan 23, 2009. Leave a Comment

Which Applications You Should Move to the Cloud

Over the past year, there has been much discussion over the cost benefits of Cloud Computing.  Companies such as Amazon, Rackspace, Google, and SalesForce all introduced Cloud platforms in 2008 that stirred our imaginations and our hopes of a more cost effective and efficient way to deliver Web applications. By virtue of the amount of vendor activity and general buzz, it is pretty clear that Cloud Computing may be the right delivery platform given our current economic environment.  That said, it also has its critics, who site that using the Cloud as a platform for “real world” enterprise applications has some issues.  Many believe that it is unproven in the areas of data security and platform stability and reliability.  But even to critics it remains a compelling alternative.   So, if Cloud Computing is at the very least “a” destination platform for delivering Web applications…what types of applications make the most sense to make the journey to the Cloud?

Cloud Testing vendor SOASTA, has been in a very unique, first-hand position of observing how early adopters of the Cloud have made their decision as to which applications they would move and why.   For the past six months, SOASTA has been Cloud Testing hundreds of Web applications from a wide variety of industries such as consumer products, mobile, social networking, and financial services.   These companies range from small start-ups to the enterprise (i.e. Hallmark, Genentech, Proctor & Gamble, Qtrax, Pelago)—all seeking ways to reduce costs of delivery by using the Cloud.

From this experience, we see the following types of applications as the forerunners in moving to the Cloud:

Sales & Marketing Applications: these are applications that center around a one-time activity such as a marketing campaign for a new product or the preparation for a holiday (i.e. an influx of e-cards or flower delivery on Valentine’s Day).  A recent event that could have benefitted from Cloud Testing was the US Presidential Inauguration. Hundreds of millions of people turned to the Internet to watch the event, causing many site outages or failures.  The characteristics of this type of Web application that make them Cloud-worthy are:
a.    Event-driven, tied to a specific time period or date
b.    Large Load (Scale), Web traffic is unknown and must be met
c.    Variable Load, prone to web traffic spikes or surges
d.    Global, used by a widely distributed audience
e.    Media Rich, employ a high level of dynamic content such as Ajax or Flash and subsequently may require more compute power

Composite Applications: these are applications that typically aggregate data services such as RSS feeds from many sources into one application. Common composite applications include Priceline, NASDAQ and Facebook. The Cloud has proven to be a very a low cost, easy-to-use, aggregation and deployment platform.

Collaborative Applications: these are applications that have many of the same attributes the previous two applications.  These are typically shared or group applications with high potential for scale and spikes in Web traffic, global users, and dynamic content. They often require shared access and availability to large amounts of compute power. Because they are not typically revenue-generating applications, they are ideally suited for a low cost delivery platform such as the Cloud.  Salesforce.com, eBay, Youtube, and some Wikis, fit this category.

The final analysis suggests that every application has unique deployment requirements and that Cloud Computing offers many companies a low cost alternative.  SOASTA’s own application, CloudTest, leverages Cloud Computing to simulate real world Web traffic for testing Web applications and networks. Testing has less of a requirement for secure or portable data than most transaction-oriented applications. It negates some of the common concerns and is ideally suited for the Cloud.  In the end, companies should choose the deployment platform that fits their application’s specific requirements and not judge every application as having the same requirements.

Cloud Computing is quickly establishing itself as a viable delivery platform. The real question remains…for which applications?  Proving once again that it’s not the destination that matters, it’s the journey.   Let the journey begin.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Web Performance is Customer Service

Posted by Tom Lounibos on Jan 23, 2009. Leave a Comment

Cloud Testing is a Solution to the Problem


Fact, there are over 162 million registered Web sites today, yet only 15% (estimated) of them have had any actual Load Testing done to them. Which means all of us consumers are being used as the digital version of “Crash Test Dummies”. You have to ask yourself . . . doesn’t service matter to Web 2.0 companies?

Surely, companies like Amazon, FaceBook, Apple, and Netflix care about their Internet customers as much or more than traditional storefront companies do. After all, most of the time these are their “only” customers.

So why would they risk a customer experience that leaves a potential user/customer without service for hours, or perhaps worse waiting for 20 seconds as a new page is downloaded? Every year Business Week publishes its Annual List of “Top 10 Companies for Customer Service”.  Lists like these typically highlight traditional retail companies, and focus their evaluations on the performance of their sales or customer support telephone staffs. But, as we all move to the Internet to buy our goods and services, what happens to customer service on the Internet?

Given that the Internet is largely a self-service business, you may question just what  “good service” is to a Web user. Latency may be one of the barometers as to what defines what we should expect as good service from our Web sites. A recent report states that Amazon estimates it loses 1% in sales if their site experiences more then 100ms in latency. In another report Google states it loses up to 20% of its traffic to a page if that page takes more then a .5 seconds to load. Then there are actual web site crashes . . . companies like Skype have had very public crashes which there site crippled for hours.

So why are these sites failing, haven’t they been tested? Well, yes and no. They have been tested functionally, but very few companies actually do stress testing of their Web sites and applications. The problem is that it has become virtually impossible and too expensive to simulate or exceed real-world Web traffic in an internal test lab. Thus, Load Testing of Web sites, if done at all, is done in a very limited way (i.e. on average companies test less then 10% of there expected web traffic). Which means for all of us Internet surfers, strap on your Crash Helmets, and prepare to be frustrated with your carrier, your device, and with your service provider as you’re about to become the main resource testing how well their web sites perform.

But don’t despair too long. A new way to simulate real world scenarios is here and it’s called Cloud Testing. This new way of testing Web applications and networks leverages Cloud Computing to simulate Web traffic, and does so at a fraction of the cost of traditional testing methods. What used to cost millions of dollars now costs a few thousand for a few hours of testing. It’s fast, easy, and affordable and provides great insight to how applications will actually run in the real world. n the future there should be a new list of “Top Services Companies”. These companies should be recognized for as having the best customer service record for the Internet. That will be largely tied to their web performance. I suspect the winners will be using Cloud Testing.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Email Us!
Subscribe to our Feed!
Join us on LinkedIn
Find us on Facebook
Follow our Tweets
See our pics