Tag Archive for ‘load testing’ rss

Now is the Time to be in Performance Engineering!

Posted by Fred Beringer on Jun 09, 2011. 2 Comments

I’ve always considered performance engineering as the most rewarding discipline in software testing. In my opinion, this is where you have the most opportunity to learn, especially technically. Great performance engineers follow Cem Kaner principles described in his Bug Advocacy paper and especially this one:

The best tester isn’t the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.

It’s about finding the right ways to communicate problems and giving as much useful information to the developers, DBA and IT guys responsible for the infrastructure where the application under test resides. It’s about dealing with objections from these people, motivating them to consider the problem seriously and to start investigating it. It’s also about pinpointing the problem in the right direction. Great Performance Engineers need to be good salesmen and need an amazing amount of knowledge to get the issue they’ve found fixed, whether it’s in the application code, the infrastructure in which the application resides or elsewhere in the overall architecture!

Great Performance Engineers get to learn about:

  • The intricacies of load balancers, especially since they’re one of the primary sources of contention when dealing with high volume applications. A lot of companies take load balancer configuration for granted and don’t bother testing their algorithm under load.  A BIG mistake!
  • CDN configuration. Again one of the top problems our Performance Engineers find when testing applications from outside the firewall.
  • Bandwidth usage and its implication on the overall performance of the application.
  • Auto-scaling mechanisms.
  • Garbage collection, memory leaks, unoptimized database schema and queries, optimizing CPU consumption, etc.
  • Everything about front-end optimization: Browser caching, expired headers, cache busters, image optimization, lazy loading, progressive rendering, etc.

Performance Engineers are able to test today at a scale they couldn’t dream about 4 years ago. Look at the test below: a 58 min test with 7 Terabytes of data received! A “big data” problem Performance Engineers can have fun with these days.

They’re able to test from inside and outside the firewall, providing coverage for problems they couldn’t previously replicate. They can, with CloudTest, get performance results in real-time and have conversations with developers, DBAs, Ops and other IT constituents during the test, increasing their chance to solve problems quickly, and to learn. A recent engagement with a large telecommunications company in the US brought 90 people together during the 2 hour test. A great learning opportunity!

If you’re eager to learn, and help companies get the best performance from their application, this is the best time to be in performance engineering. Best of all, SOASTA is hiring!

What Keeps a 100-Year-Old Company Feeling Like a Teenager?

Posted by Tom Lounibos on Jun 06, 2011. Leave a Comment

This year IBM turns 100 years old, and last week IBM’s “Watson” was named 2011 “Person of the Year” by the Webby Awards. The mere thought of IBM as a start-up (circa 1911) boggles my mind, especially when you consider that this year they reached their all-time high water mark at $205B in market valuation. This feat is even more amazing given the somewhat rocky road that they traveled during the late 80’s and 90’s. Over the last ten years under the watchful eye of Sam Palisamo, IBM is beginning to experience a rebirth. While they are nowhere near the dominant leadership position that they held over the technology sector from 1940 – 1980 when the market was defined as “IBM and the Seven Dwarfs,” today they are beginning to show signs of re-emergence as — if nothing else — the “supervising adult ” that their 100 years of existence entitles them to.

So what keeps an old company relevant after all these (100) years? Probably the same things that keep older people young…they hang out with their kids or grandchildren. In IBM’s case they have learned (over the years) to partner with a few young, innovative companies that are proving to be the new “game changers” in several traditional markets — even in some markets that have long been considered to be IBM’s strongholds. By partnering up with these young “upstarts,” IBM has given their customers fresh alternatives for new technologies and approaches for dealing with a rapidly changing business world. Perhaps even more importantly for many of their customers, IBM is also delivering a much-needed layer of “adult supervision” in this increasingly crowded and complex vendor landscape. Their years of experience enable IBM to become a trusted advisor to their customers on how to navigate through this vendor mine field.

As one those lucky few young upstarts that Grandpa IBM has chosen to partner with, we here at SOASTA get the advantage of their many years of experience surviving and thriving in both up and down markets. We also get to benefit from what may be the greatest technology distribution channel ever compiled, a channel that no start-up could ever replicate organically.

Time will only tell if Grandpa IBM and its young upstart partner SOASTA will make an interesting combination for the IBM nation, but the hourglass has been turned. One thing is for sure, SOASTA is one youngster that is eager to learn from Grandpa.

NEW Web Services Performance Certification Program

Posted by Tom Lounibos on Jul 20, 2009. 1 Comment

I’m very excited to announce the new SOASTA Performance Certification Program designed to enable companies deploying software in the Cloud, at hosted data centers, or behind corporate firewalls to certify that their websites have been tested and have met or exceeded industry benchmarks for performance at peak levels of user traffic.  For the past ten years, the dirty little secret in the web development  community has been that whether due to cost, complexity or lack of resources, the vast majority of web applications and sites have not been tested at normal user volumes, much less for unexpected spikes in traffic. Which means our user communities have become the testers for virtually every website, a risk that has proven very costly time and again.

Times are changing–for every new service there are at least five competitors. Having a website that is slow to download or crashes frequently is no longer be tolerated.  “To our customers, performance matters!” said Lew Moorman, President, Rackspace Cloud. “By utilizing the SOASTA Performance Certification program our customers will have the ability to isolate performance issues before they occur by simulating real world user activity and traffic. Performance certification ensures that websites are being tested, which only leads to greater levels of reliability.”

Supporting SOASTA in this performance certification initiative are industry leaders in cloud computing including platform vendors, testing companies and independent cloud service providers. They include 3Tera, Appistry, Chegg.com, Enomaly, GoGrid, Hexaware Technologies, Intuit, JackBe, PowerTest, Rackspace, Rightscale, rPath and Zephyr. Concerns over performance and reliability are consistent themes heard by each of these companies when their customers face the uncertainties of extending access to applications outside the firewall, not to mention moving those applications completely offsite. This diverse group of companies represents all aspects of the application lifecycle, from development to deployment, all supporting the need for independent, external validation of end-to-end performance.

Many companies are currently considering or have already moved their applications to the Cloud, typically to lower costs and/or to take advantage of the elasticity of the compute power offered by the Cloud. However, just as in the data center, simply adding low cost servers doesn’t solve most performance problems. Performance-related issues, such as latency or an actual website crash, are often the result of a change to the deployment environment, changes in the application itself, or the inclusion of third-party content. The only way to certify a site’s performance is by simulating both expected and potential peak traffic while monitoring the impact. With Performance Certification, this is no longer too expensive or too complex to achieve and alleviates additional strain on internal resources.

“Improving performance cannot be achieved solely by adding hardware,” said Michael Crandell, CEO, Rightscale. “Web applications and deployment stacks are complex and must be tested under real world traffic conditions to assess what actual performance will be in production. Having a certification process that ensures sites are tested is an essential element of cloud computing development.”

We are very excited about this new program to certify websites and application performance! We are particularly appreciative of all of our partners who also feel that “Performance Matters” in delivering the best possible user experience.

SOASTA Performance Certification Program

Posted by Tom Lounibos on Jul 20, 2009. 2 Comments

SOASTA Performance Certification is designed as a turnkey process, minimizing disruption by eliminating any unnecessary impact on the existing environment. SOASTA and its partners collaborate with companies to define common use cases and then simulate those scenarios in the most accurate way possible—by using the web to test the web.

There are a number of purpose built tools to analyze the performance characteristics of individual components of a web-based application, such as web page design, application design, database implementation, or network architecture. SOASTA’s Performance Certification leverages its Global CloudTest™ Platform to provide an affordable end-to-end analysis of a site’s performance as well as measure responsiveness at normal and peak usage levels. Certified sites receive a comprehensive report on Key Performance Indicators (KPIs), with a focus on response times achieved at various user loads.

Certification confirms that the site has been tested at specific traffic volume levels (1K, 10K, 25K, 50K, or 100K users) and has been measured against KPIs such as response time and error rates. In addition, as part of the certification process SOASTA and its partners provide valuable analysis of site latency to help companies improve the overall performance and responsiveness of their website.

Companies such as Chegg.com, Dell, Intuit and Vovici are just a few of the companies that have had their websites SOASTA Certified. “As traffic to our website continues to grow at record levels, we need to ensure that our site will be able to meet the demand,” said Dan Bartow, Manager of Performance Engineering, Intuit TurboTax. “SOASTA enables us to simulate every possible user scenario including massive load. Thanks to SOASTA’s Performance Certification process, we are confident that our site will be ready for almost any situation we might encounter.”

The First Cloud-Based Global Test Platform

Posted by Tom Lounibos on Feb 17, 2009. 2 Comments

For the past several months we (SOASTA) have been performing global load and performance tests by leveraging multiple cloud platforms (cross-cloud testing) to generate and simulate Web traffic originating from around the world. Up until last November we had been limited to using Amazon’s New Jersey locations. Now we have access to and can generate traffic (load) from 15 locations around the globe. For the several customers that have experienced it, the results have been amazing. While simulating global traffic is not all together new, using cloud computing makes it, for the first time, fast, scalable, and affordable. Tomorrow we will announce the availability of The First Cloud-Based Global Test Platform. This New Test Platform is NOW available to all corporations around the world for Global Load testing on there web applications.   Prices starts at $1,000/Test Hour.

One of our customers QTRAX recently ran a test that simulated several million users coming from London, Hong Kong, New York, and Los Angeles. It took less then 30 minutes to provision the entire test environment (hardware & software) at a cost of less than $3,000 to perform the test. Chris Roe, CTO of QTRAX, stated, “Our customer are located all around the world, and not just here in New York.  So we needed a test platform that could best simulate our every day traffic.  SOASTA pulling together multiple Cloud Platforms (cross-cloud testing)to deliver a very “elastic” test platform, was a perfect fit for testing our site.” Cloud Testing is changing how we test forever.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Traditional Vendors Out, New Players Emerge in Cloud Testing

Posted by Tom Lounibos on Jan 30, 2009. Leave a Comment

Every recession over the past 30 years has proven to be a forcing function for significant technological change.  In good times, corporations can get a bit careless or lose focus. However, in bad times, corporations go to great lengths to find new levels of cost efficiency in their operations . . . even if it means changing processes and technology that they had used for many years.  This recession is no different, change is beginning to happen, and traditional software vendors are the prime targets. The traditional “License” model employed by traditional software vendors is officially dead with this recession. The “Software as a Service” business model or even more specifically, the “Pay for Use” model, have emerged as clear winners.  IDC recently raised its 2009 projections for the SaaS market. They now expect this segment to grow by more than 40% this year.  This business model shift, coupled with the emergence of the new deployment platform of Cloud Computing will make it very difficult for traditional vendors to maintain their thrones.

Take my business, the test marketplace.  Our market was long dominated by Mercury Interactive which was bought by HP several years ago.   Mercury’s LoadRunner was the ideal tool for testing client-server applications in the late 80′s and 90′s, and its success drove Mercury to its leadership position under a license-based model.  However, a license model that was very expensive ($30k/Test Hour). Today, the world has changed for HP/Mercury. We are all now developing and deploying new and much more dynamic Web applications for consumers around the world. Mercury’s technology, business, and deployment model are looking a bit like Tom Jones performing at a Beyonce concert . . . a little out of place. New test vendors are beginning to emerge as the new leaders of the Cloud Testing market.  This includes new players like SOASTA that deliver a “pay only for test time used” ($1k/Test Hour).  Companies like these are greatly reducing the cost of testing while even enabling more and better quality testing. This is an example of another changing of the guard, made possible by another down economic period and a requirement for greater reliability.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Genentech Tests Cloud Computing

Posted by Tom Lounibos on Jan 29, 2009. Leave a Comment

As Michael Liedtke of Associated Press reported last week, and subsequently now we can also report, that indeed Genentech has migrated its 16,000 employee’s to Google Apps for its email and for some office suite applications. However, the more interesting back story to this article was before they made the decision, Genentech wanted to ensure that Google would scale. Google . . . scale?  That’s right they questioned what kind of user experience there employees would have when accessing applications deployed in a cloud. To get comfortable, Genentech decided to do some Cloud Testing using SOASTA’s new On Demand CloudTest Service. Over the course of a couple weeks Genentech were able to test several different user scenarios including having 16,000 simulated Genentech employees hitting Google Calendar at the same instant.  The testing went well, and the rest is history. Genentech has dipped their big toe into the Cloud Landscape, and by all the feedback we have received,  it went very well.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Cloud Computing is Changing How we Test Web Sites Forever!

Posted by Tom Lounibos on Jan 28, 2009. Leave a Comment

Okay, I know I’m biased, but as the rest of the blogging world focuses their attention on the “Cloud Platform Wars”, debating endlessly on which Cloud platform vendor will emerge as the “winner” in the coming years, quietly, several companies (from young start-ups to enterprise class) are leveraging the access, availability and affordability of Cloud Computing for testing their Web sites.  Even more amazingly, they are testing without even considering which Cloud platform they are using.  They are Cloud Testing!

Load, Performance, and Stress testing Web sites has been a black art of Web development shops for years. . . requiring a huge amount of compute power, expertise, and expensive system and test software. The total cost of Performance testing has reached an estimated $30,000/test hour. . . making testing costs prohibitive for most companies. That is, until Cloud Testing emerged in 2008.

Now companies are leveraging the limitless power of Cloud Computing to simulate Web (scale) traffic to test their sites before going into LIVE production. They are greatly reducing the occurrences of performance related errors, latency, and actual site crashes while reducing the cost of testing down to $1,000/test hour.

So, while the rest of the world tries to figure out whether or not Cloud Computing is for real, we are seeing many customers seeking a higher level of reliability for their web sites,  at a lower cost, finding a real value proposition in Cloud Computing.

Cloud Testing is Changing How we Test Web Sites Forever!

contact me at: tlounibos@soasta.com; twitter.com/lounibos

SOASTA Performs 500th Cloud Test

Posted by Tom Lounibos on Jan 27, 2009. Leave a Comment

It seems like ten years have past since our first Cloud Test, but last night we hit the 500th test milestone with a test for Hallmark.com.   An amazing stress test that lasted a little over an hour and simulated over 600,000 users hitting their site.  The analytics (1 TB) were amazing in their detail of how the site and network would perform in live production.   Looking back at our first test,  we were so inexperienced in this new method of testing.  We had no idea what Cloud Computing would bring to Web testing.   It has been an amazing year of discovery and experience.  In the end, we once again re-discovered the point of testing is not in the ability to run a test….but the real point of testing is in the Results and Analytics that the test produces. Cloud Testing is changing testing forever. Thank you to all of our very loyal customers!

contact me at: tlounibos@soasta.com; twitter.com/lounibos

If Cloud Computing is The New Destination, Then Where does the Journey Begin?

Posted by Tom Lounibos on Jan 23, 2009. Leave a Comment

Which Applications You Should Move to the Cloud

Over the past year, there has been much discussion over the cost benefits of Cloud Computing.  Companies such as Amazon, Rackspace, Google, and SalesForce all introduced Cloud platforms in 2008 that stirred our imaginations and our hopes of a more cost effective and efficient way to deliver Web applications. By virtue of the amount of vendor activity and general buzz, it is pretty clear that Cloud Computing may be the right delivery platform given our current economic environment.  That said, it also has its critics, who site that using the Cloud as a platform for “real world” enterprise applications has some issues.  Many believe that it is unproven in the areas of data security and platform stability and reliability.  But even to critics it remains a compelling alternative.   So, if Cloud Computing is at the very least “a” destination platform for delivering Web applications…what types of applications make the most sense to make the journey to the Cloud?

Cloud Testing vendor SOASTA, has been in a very unique, first-hand position of observing how early adopters of the Cloud have made their decision as to which applications they would move and why.   For the past six months, SOASTA has been Cloud Testing hundreds of Web applications from a wide variety of industries such as consumer products, mobile, social networking, and financial services.   These companies range from small start-ups to the enterprise (i.e. Hallmark, Genentech, Proctor & Gamble, Qtrax, Pelago)—all seeking ways to reduce costs of delivery by using the Cloud.

From this experience, we see the following types of applications as the forerunners in moving to the Cloud:

Sales & Marketing Applications: these are applications that center around a one-time activity such as a marketing campaign for a new product or the preparation for a holiday (i.e. an influx of e-cards or flower delivery on Valentine’s Day).  A recent event that could have benefitted from Cloud Testing was the US Presidential Inauguration. Hundreds of millions of people turned to the Internet to watch the event, causing many site outages or failures.  The characteristics of this type of Web application that make them Cloud-worthy are:
a.    Event-driven, tied to a specific time period or date
b.    Large Load (Scale), Web traffic is unknown and must be met
c.    Variable Load, prone to web traffic spikes or surges
d.    Global, used by a widely distributed audience
e.    Media Rich, employ a high level of dynamic content such as Ajax or Flash and subsequently may require more compute power

Composite Applications: these are applications that typically aggregate data services such as RSS feeds from many sources into one application. Common composite applications include Priceline, NASDAQ and Facebook. The Cloud has proven to be a very a low cost, easy-to-use, aggregation and deployment platform.

Collaborative Applications: these are applications that have many of the same attributes the previous two applications.  These are typically shared or group applications with high potential for scale and spikes in Web traffic, global users, and dynamic content. They often require shared access and availability to large amounts of compute power. Because they are not typically revenue-generating applications, they are ideally suited for a low cost delivery platform such as the Cloud.  Salesforce.com, eBay, Youtube, and some Wikis, fit this category.

The final analysis suggests that every application has unique deployment requirements and that Cloud Computing offers many companies a low cost alternative.  SOASTA’s own application, CloudTest, leverages Cloud Computing to simulate real world Web traffic for testing Web applications and networks. Testing has less of a requirement for secure or portable data than most transaction-oriented applications. It negates some of the common concerns and is ideally suited for the Cloud.  In the end, companies should choose the deployment platform that fits their application’s specific requirements and not judge every application as having the same requirements.

Cloud Computing is quickly establishing itself as a viable delivery platform. The real question remains…for which applications?  Proving once again that it’s not the destination that matters, it’s the journey.   Let the journey begin.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Email Us!
Subscribe to our Feed!
Join us on LinkedIn
Find us on Facebook
Follow our Tweets
See our pics