Tag Archive for ‘performance testing’ rss

“Minority Report” Defining Today’s User Interface

Posted by Tom Lounibos on Jan 18, 2012. Leave a Comment

Think back to Steven Spielberg’s 2002 classic, “Minority Report”, the movie that explored the future of human/machine interface in 2054. It was both cool and exciting, and it was also years away from reality. Or was it? Here we are, now ten years removed, and some of these concepts, gesture-based applications (everywhere), hovering gestures (Apple patent), 3D navigation (everywhere), and voice (Siri), are very much real.

John Underkoffler is the UI designer who built the technology for Minority Report and in real life, too. His 15-minute TED presentation is well worth watching.

From logistics and supply chain to financial services, new applications evolve every single day to leverage advancements in man and machine user interfaces. In many cases there is simply no other option than to rapidly wade through the enormous amounts of data these applications generate in search of the actionable intelligence.

One of these Big Data issues occurs while performance testing web and mobile applications. These applications simulate millions of people around the globe hitting a web site or mobile application to provide predictive analysis as to where performance bottlenecks may occur. In doing so, they generate terabytes of performance-related, real-time data. Analysis of this data is extremely difficult using traditional, one-dimensional test tools because problems are often buried amongst layers and layers of data.

Over the past few years, companies such as Splunk and SOASTA have emerged because their multi-dimensional capabilities enable testers to see several pieces of performance data on the same timeline and in a single view. These kinds of user interfaces allow for real-time resolution to problems such as performance for a growing number of leading companies.

Of course new problems arise as these problems are solved. The next generation will surely struggle with delivery of a stable user experience when application UIs are gesture-based, voice-based or even 3D. I will leave that question for another day. The bottom line is this: Minority Report has arrived. Is your team ready?

 

Finding Bottlenecks Through Monitoring and Iterative Load Tests

Posted by Yuriy Verkhovoy on Jan 05, 2012. Leave a Comment

A medium-size web development shop engaged us for its first experience with external, scale testing on a brand new application scheduled to roll out a few months after we started. The application is written in Ruby on Rails (ROR) with a MySQL backend, and serves as an internal social networking hub for a company. Through this application, users can create projects and tasks that are both work related and personal, invite others to join them in completing these tasks, share their calendars, and exchange instant messages with other online members. The front end is entirely JavaScript driven and consists of a number of widgets, all of which are updated through AJAX calls to the server. In addition, the client intends to host this application in the cloud, which would be a first for them.

The test consisted of 10 scenarios including simple browsing of the site, creating projects and tasks, and randomly sending instant messages to other online members. The most challenging part of test creation was dealing with the asynchronous nature of the application. There were no clear-cut events that broke the script into URLs or pages. There was only one URL followed by a number of AJAX calls to the server. CloudTest’s nested transaction grouping allowed us to put these requests into logical groups that represented all of the main business processes to be tested.

The first test was against a single server that contained all of the moving parts of the application and was hosted onsite. The application performed well until we reached 200 simultaneous virtual users. At that point, things got really slow. At about 300 users, the site was unusable and every request came back with a HTTP 500 error.

Our second test was against a more distributed environment. The application server and the database were split between two different machines, which were hosted in Amazon EC2. The site did slightly better, topping out at 300 users. Using CloudTest’s monitoring, we noticed that the database was not working very hard and that the web server was at nearly 100% CPU utilization for most of the test.

During subsequent tests we tried a variety of configurations. CloudTest makes it very easy to do iterative load testing. We spread the load even more by putting several application servers behind an EC2 load balancer and increasing the horsepower of the database server. Although most of these changes made a visible improvement, we were still faced with an obvious bottleneck. At roughly 800 simultaneous users, the application started to slow down, and at 1200 users it became almost unusable. At this point all of the usual suspects looked innocent. The app servers’ CPU never got higher than 50%, the database had plenty of breathing room and we were not limited by bandwidth.

When we looked at the “Collection Summary” report from CloudTest we realized that one particular transaction, which was taking longer to complete than others, needed more attention during the test. As you can see from the chart below, the “CREATE TASK” transactions took significantly more time to complete, even when there was little load on the system.

The ops team monitored the app server with New Relic, which has a ROR profiling module and is integrated into the SOASTA dashboards. We found that a CREATE TASK business process was using a ROR ORM Engine called Active Record to perform MySQL queries. A particular query in the CREATE TASK process was not “tuned”, and instead of running one or two SQL statements to retrieve the data, hundreds of queries were run each time a list of tasks was requested by the application. Although the queries themselves were very lightweight, running hundreds of them in sequence for each process slowed things down considerably.

Our findings were brought to the attention of the development team. The team was able to revise the process to access the same data with just one query to the database. As a result, in our subsequent tests we saw a great improvement in the CREATE TASK transaction. Monitoring the infrastructure during a load test once again proved invaluable, and having the results on the same timeline helped surface the issues much more quickly.

 

The Future of Testing: Evolve or Elimination. It’s a Choice.

Posted by Tom Lounibos on Nov 17, 2011. 1 Comment

I’ve been seeing a lot of tweets and blogging around the future role of the Software Tester.  Some have suggested that, with all the recent advancements made in test automation, the test community may face employment challenges. I strongly disagree. That said, rapid technology innovation necessitates that users adapt and evolve, and the test community will not be immune to this requirement.

The fact is that today, in 2011, the majority of Testers are still relying on the same tools that they used in 1989. How is it that this community has been not been forced to change all that much over the past 20 years? This is highly unusual and counterintuitive in an industry that has long been affected by term limits for technology, one that is known for discarding old tools in favor of new tools to support emerging technologies.

How is it that so many Testers still use tools borne during the Reagan Era and when client server applications dominated the tech landscape? Have our applications neglected to evolve? Of course not!

One theory is that Testers are just wired differently, that they are more resistant to the forces of change. If this is the case, consumers beware: the impact will surely lead to an increase in web and mobile app failures. Why? If Testers hold onto previous testing practices then today’s dynamic applications will need more and more frequent performance testing because of the demands for speed, scale or cost savings.

So what needs to happen?

Like other professionals, Testers must evolve their skill sets. They must acknowledge and respond to the pressure to deliver actionable intelligence back to the business as a result of their testing. Further, they must begin to understand and accept responsibility for all aspects of delivering performance. Performance is the correlation of Application, Infrastructure, Network and Users, and a valuable, savvy Tester will deliver aggregated and correlated results to support their actionable intelligence.

It’s a new world of HTML, gesture-based UOs, Cloud, streaming video and voice, and Testers must embrace the opportunity or they will find themselves out of work. Carpe Diem Testing. Adapt and Evolve. Join me in helping the testing community become part of the cognoscenti, be on the forefront of this technology adoption lifecycle, earn industry respect by embracing testing’s place in the revenue cycle, and be the leaders in delivering the next generation of High Performance Applications.

 

Now is the Time to be in Performance Engineering!

Posted by Fred Beringer on Jun 09, 2011. 2 Comments

I’ve always considered performance engineering as the most rewarding discipline in software testing. In my opinion, this is where you have the most opportunity to learn, especially technically. Great performance engineers follow Cem Kaner principles described in his Bug Advocacy paper and especially this one:

The best tester isn’t the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.

It’s about finding the right ways to communicate problems and giving as much useful information to the developers, DBA and IT guys responsible for the infrastructure where the application under test resides. It’s about dealing with objections from these people, motivating them to consider the problem seriously and to start investigating it. It’s also about pinpointing the problem in the right direction. Great Performance Engineers need to be good salesmen and need an amazing amount of knowledge to get the issue they’ve found fixed, whether it’s in the application code, the infrastructure in which the application resides or elsewhere in the overall architecture!

Great Performance Engineers get to learn about:

  • The intricacies of load balancers, especially since they’re one of the primary sources of contention when dealing with high volume applications. A lot of companies take load balancer configuration for granted and don’t bother testing their algorithm under load.  A BIG mistake!
  • CDN configuration. Again one of the top problems our Performance Engineers find when testing applications from outside the firewall.
  • Bandwidth usage and its implication on the overall performance of the application.
  • Auto-scaling mechanisms.
  • Garbage collection, memory leaks, unoptimized database schema and queries, optimizing CPU consumption, etc.
  • Everything about front-end optimization: Browser caching, expired headers, cache busters, image optimization, lazy loading, progressive rendering, etc.

Performance Engineers are able to test today at a scale they couldn’t dream about 4 years ago. Look at the test below: a 58 min test with 7 Terabytes of data received! A “big data” problem Performance Engineers can have fun with these days.

They’re able to test from inside and outside the firewall, providing coverage for problems they couldn’t previously replicate. They can, with CloudTest, get performance results in real-time and have conversations with developers, DBAs, Ops and other IT constituents during the test, increasing their chance to solve problems quickly, and to learn. A recent engagement with a large telecommunications company in the US brought 90 people together during the 2 hour test. A great learning opportunity!

If you’re eager to learn, and help companies get the best performance from their application, this is the best time to be in performance engineering. Best of all, SOASTA is hiring!

What Keeps a 100-Year-Old Company Feeling Like a Teenager?

Posted by Tom Lounibos on Jun 06, 2011. Leave a Comment

This year IBM turns 100 years old, and last week IBM’s “Watson” was named 2011 “Person of the Year” by the Webby Awards. The mere thought of IBM as a start-up (circa 1911) boggles my mind, especially when you consider that this year they reached their all-time high water mark at $205B in market valuation. This feat is even more amazing given the somewhat rocky road that they traveled during the late 80’s and 90’s. Over the last ten years under the watchful eye of Sam Palisamo, IBM is beginning to experience a rebirth. While they are nowhere near the dominant leadership position that they held over the technology sector from 1940 – 1980 when the market was defined as “IBM and the Seven Dwarfs,” today they are beginning to show signs of re-emergence as — if nothing else — the “supervising adult ” that their 100 years of existence entitles them to.

So what keeps an old company relevant after all these (100) years? Probably the same things that keep older people young…they hang out with their kids or grandchildren. In IBM’s case they have learned (over the years) to partner with a few young, innovative companies that are proving to be the new “game changers” in several traditional markets — even in some markets that have long been considered to be IBM’s strongholds. By partnering up with these young “upstarts,” IBM has given their customers fresh alternatives for new technologies and approaches for dealing with a rapidly changing business world. Perhaps even more importantly for many of their customers, IBM is also delivering a much-needed layer of “adult supervision” in this increasingly crowded and complex vendor landscape. Their years of experience enable IBM to become a trusted advisor to their customers on how to navigate through this vendor mine field.

As one those lucky few young upstarts that Grandpa IBM has chosen to partner with, we here at SOASTA get the advantage of their many years of experience surviving and thriving in both up and down markets. We also get to benefit from what may be the greatest technology distribution channel ever compiled, a channel that no start-up could ever replicate organically.

Time will only tell if Grandpa IBM and its young upstart partner SOASTA will make an interesting combination for the IBM nation, but the hourglass has been turned. One thing is for sure, SOASTA is one youngster that is eager to learn from Grandpa.

At SOASTA we’re in the wisdom business

Posted by Fred Beringer on Apr 07, 2011. 2 Comments

An interesting discussion with a former LoadRunner Product Manager piqued my interest and triggered this post. The real trigger was this part: CloudTest (like others, including LoadRunner) is still just a testing solution and as such serves as a filter or gate-keeping function in the lifecycle. [attrib: Jim Duggan, Gartner]. I’m gonna disagree on that one as I’ve had a different opinion on the “gate” part for the past 15 years. In my opinion, the goal of performance testing (and software testing in general) is not to be used as a gate. Testing is only one part of the equation when you need to decide whether or not to go live. There are other important factors to take into consideration, and depending on the context and timing they might be more important. The marketing and sales side of the software industry often prevails, for better of for worse.

The true goal of Performance Testing is to provide wisdom. Maybe the sole purpose of LoadRunner is to be a testing solution that serves as a gate, but at SOASTA we’re in the wisdom business.


The first objective of performance testing is to gather relevant data from EVERY components involved in the overall performance of your application. This is your raw and dumb data.

  • Data from your application itself, memory, CPU consumptions, number of processes, heapsize, etc.
  • Data from the application’s underlying infrastructure: Application servers, web servers, databases, SSL servers, CMS, memcache servers etc.
  • Data from your application’s ecosystem: CDNs, Load balancers, switches, routers, DNS, etc.

You end up with LOTS of data. To give you an idea of the magnitude of data you might gather, one of the tests we’ve performed with a major TV network generated over 7 terabytes of data in less than one hour. Transfer was 17GigaBit per second! Thanks to CloudTest we’re able to gather these terabytes of data in real-time. But they’re USELESS data if you don’t have a mechanism to transform them into information. In order to deal with this BIG DATA PROBLEM you need a real-time, in memory OLAP engine. Gartner predicts that By 2014, 30 percent of analytic applications will use in-memory functions to add scale and computational speed. It looks like SOASTA is ahead of the curve!

Information is created by analyzing relationships and connections between data. Information should help you make some sense from your data and become relevant to your business:

  • What’s the relationship between the number of virtual users and the memory consumed by my application server?
  • How much server capacity am I left with when I reach 10,000 concurrent users during my test?
  • What is the correlation between the number of process counts on the database server and the overall throughput?
  • At what stage during the test do I see a drop in overall response time? Can I correlate this drop with another data to understand this behavior?
  • What is the correlation between an increase in response time and the number of errors coming from my SSL Server?
  • Why is 90% of response time for my overall homepage taken by this particular file? Where does it come from? Why does it take longer than the other page assets?

By combining, correlating and aggregating your data you’re able to build enough information to understand the behavior of your application and its entire ecosystem.

Knowledge is created by receiving, absorbing and understanding the information. From this knowledge we can make decisions and take actions. If you’re observing a surge of traffic going into one particular webserver, impacting the overall response time for some of your visitors, you can pinpoint the problem to a misconfiguration in one of your load balancer. You’ve got sufficient knowledge to take action and make a change on the fly. That’s what we call actionable intelligence at SOASTA. And with modern testing tools, such as CloudTest, it can happen in real-time. That’s the agile way of doing performance testing!

Wisdom deals with the future and predictability.  This when you apply knowledge to change your process and reach your true objectives. This is the level where you test your predictions, execute them, monitor the results and adapt. As an example, what if you’re expecting 200,000 concurrent users on your website after a big product announcement? You think your visitors will behave in a particular way (you’ve got historical data to back this up): 50% browsing the site, 15% login, 15% putting items in and out of the shopping cart and 20% watching a video. How much bandwidth do you need? What should be the right configuration for your load balancer? What assets should be within the CDN? What should be the configuration of the memcache server? What if one of the web server crash? What would be the impact for your visitors? What would be the impact on your business?

Today, a lot of companies stop at the information stage and analyze their metrics retroactively to understand what has happened. There are some behavioral reasons, but most of them are just not equipped to reach further levels. You need a mechanism to:

  • Gather raw data in an efficient manner.
  • Build information by combining, correlating and aggregating these data to start to understand behavior.
  • Bring you knowledge, in real-time, so you can make decisions and take actions, FAST.
  • Perform predictive analysis, build prediction models so you can help your business and your customers in the best possible way. Gartner predics that by 2014, 30 percent of analytic applications will use proactive, predictive and forecasting capabilities. SOASTA is already there!

At SOASTA we’re not in the gate business. We’re in the wisdom business.

What Happens When Vendors Repackage Old Technology and Call it a Cloud Service?

Posted by Tom Lounibos on Oct 23, 2009. Leave a Comment

In an effort to remain relevant, some software vendors take marketing advantage of the newest hot technology fad at the expense of their own customers. Cloud computing (the newest hot trend) has definitely been defined and positioned by some traditional software and hardware vendor’s marketing organizations to meet their specific agenda, which usually means extending the life of an existing product or product line.  It has become a virtual “cloud rush” as to how many times “cloud computing” can be mentioned in product and marketing collateral. . . including collateral for products that were first developed back when Bill Clinton was president!

A great example of this “cloud rush” is Hewlett Packard with its LoadRunner product.  LoadRunner was developed in the early 90’s to help corporate development teams test client/server applications.  It became, over time, the de-facto standard testing tool for most enterprise companies and was priced accordingly.  Entry-level pricing began at $100,000 and if you needed to simulate thousands of users the cost skyrocketed into the millions of dollars very quickly.

Today, HP is attempting to “perfume the pig” so to speak, by repackaging LoadRunner into a new cloud-based service called Elastic Test. To the uneducated observer it simply looks like a new cloud service for testing web applications.  The problem is that it’s the same LoadRunner product built almost 20 years ago to test client/server applications and it carries a lot of technology baggage along with it. Subsequently, HP chooses to pass along a lot of this baggage in the form of costs back to its customer base.  For example, an entry-level HP virtual test will take weeks to develop and will have a “starting” cost of $45,000.  Hardly living up to cloud computing’s value proposition for on-demand services that provide ease, speed and affordability.

Newer players are quickly swooping in to fill in the gap left by HP’s lack of R&D in the cloud space. New players such as SOASTA, whose CloudTest service was built exclusively to leverage cloud computing for testing. It offers a low-cost, highly scalable and easy-to-use on-demand test model for customers.  As a comparison, the $45,000 “entry-level” HP Virtual Test that takes weeks to perform and deliver results, can be done within 24 hours at a cost of $750 using SOASTA’s On-Demand CloudTest service. This lower cost model also delivers greater quality testing. Traditionally, you may have only performed 6-8 performance tests on a client/server application because of the cost and time required.  Now, with true cloud testing services like SOASTA CloudTest, you can perform hundreds of tests for significantly less cost and in the same amount of time it takes to perform 6-8 traditional tests. This impact is significant, delivering greater reliability in your web applications.

Customers choosing to stay with HP as their test vendor out of loyalty will continue to have to deal with a 20 year old technology trying to conform to a 2009 eco-system, as well as pay a premium of up to 45X for that loyalty.  This, in a world where websites need more and more performance testing as they become more complex and we reach higher and higher spikes in user traffic.

Cloud computing is a “game changer” for customers seeking greater levels of web reliability. It enables new, leading-edge, agile and cost-effective cloud testing services.  However, be careful of impostors claiming to offer cloud services . . . more often then not you will be able to recognize them by their price tags and by the lack of quality in the actionable intelligence they deliver.  SOASTA CloudTest is the cloud leader in delivering the highest quality performance intelligence available on the market today.

NEW Web Services Performance Certification Program

Posted by Tom Lounibos on Jul 20, 2009. 1 Comment

I’m very excited to announce the new SOASTA Performance Certification Program designed to enable companies deploying software in the Cloud, at hosted data centers, or behind corporate firewalls to certify that their websites have been tested and have met or exceeded industry benchmarks for performance at peak levels of user traffic.  For the past ten years, the dirty little secret in the web development  community has been that whether due to cost, complexity or lack of resources, the vast majority of web applications and sites have not been tested at normal user volumes, much less for unexpected spikes in traffic. Which means our user communities have become the testers for virtually every website, a risk that has proven very costly time and again.

Times are changing–for every new service there are at least five competitors. Having a website that is slow to download or crashes frequently is no longer be tolerated.  “To our customers, performance matters!” said Lew Moorman, President, Rackspace Cloud. “By utilizing the SOASTA Performance Certification program our customers will have the ability to isolate performance issues before they occur by simulating real world user activity and traffic. Performance certification ensures that websites are being tested, which only leads to greater levels of reliability.”

Supporting SOASTA in this performance certification initiative are industry leaders in cloud computing including platform vendors, testing companies and independent cloud service providers. They include 3Tera, Appistry, Chegg.com, Enomaly, GoGrid, Hexaware Technologies, Intuit, JackBe, PowerTest, Rackspace, Rightscale, rPath and Zephyr. Concerns over performance and reliability are consistent themes heard by each of these companies when their customers face the uncertainties of extending access to applications outside the firewall, not to mention moving those applications completely offsite. This diverse group of companies represents all aspects of the application lifecycle, from development to deployment, all supporting the need for independent, external validation of end-to-end performance.

Many companies are currently considering or have already moved their applications to the Cloud, typically to lower costs and/or to take advantage of the elasticity of the compute power offered by the Cloud. However, just as in the data center, simply adding low cost servers doesn’t solve most performance problems. Performance-related issues, such as latency or an actual website crash, are often the result of a change to the deployment environment, changes in the application itself, or the inclusion of third-party content. The only way to certify a site’s performance is by simulating both expected and potential peak traffic while monitoring the impact. With Performance Certification, this is no longer too expensive or too complex to achieve and alleviates additional strain on internal resources.

“Improving performance cannot be achieved solely by adding hardware,” said Michael Crandell, CEO, Rightscale. “Web applications and deployment stacks are complex and must be tested under real world traffic conditions to assess what actual performance will be in production. Having a certification process that ensures sites are tested is an essential element of cloud computing development.”

We are very excited about this new program to certify websites and application performance! We are particularly appreciative of all of our partners who also feel that “Performance Matters” in delivering the best possible user experience.

SOASTA Performance Certification Program

Posted by Tom Lounibos on Jul 20, 2009. 2 Comments

SOASTA Performance Certification is designed as a turnkey process, minimizing disruption by eliminating any unnecessary impact on the existing environment. SOASTA and its partners collaborate with companies to define common use cases and then simulate those scenarios in the most accurate way possible—by using the web to test the web.

There are a number of purpose built tools to analyze the performance characteristics of individual components of a web-based application, such as web page design, application design, database implementation, or network architecture. SOASTA’s Performance Certification leverages its Global CloudTest™ Platform to provide an affordable end-to-end analysis of a site’s performance as well as measure responsiveness at normal and peak usage levels. Certified sites receive a comprehensive report on Key Performance Indicators (KPIs), with a focus on response times achieved at various user loads.

Certification confirms that the site has been tested at specific traffic volume levels (1K, 10K, 25K, 50K, or 100K users) and has been measured against KPIs such as response time and error rates. In addition, as part of the certification process SOASTA and its partners provide valuable analysis of site latency to help companies improve the overall performance and responsiveness of their website.

Companies such as Chegg.com, Dell, Intuit and Vovici are just a few of the companies that have had their websites SOASTA Certified. “As traffic to our website continues to grow at record levels, we need to ensure that our site will be able to meet the demand,” said Dan Bartow, Manager of Performance Engineering, Intuit TurboTax. “SOASTA enables us to simulate every possible user scenario including massive load. Thanks to SOASTA’s Performance Certification process, we are confident that our site will be ready for almost any situation we might encounter.”

CLOUD: Performance Intelligence Delivers Certainty

Posted by Tom Lounibos on Mar 04, 2009. Leave a Comment

Tom Wailgum wrote an interesting piece in CIO Magazine this past weekend about enterprises “still crazy for BI after all these years”.  He notes that  companies still see business intelligence as transformational, but he also notes, many execs are still relying on their gut to make critical business decisions.  Take web site deployment for example. Even though web usage has risen to amazing levels in recent years and sites have become every organizations’ primary sales and marketing channel, we still push out new sites every day without a clue as to how they will perform in LIVE production. Don’t get me wrong, we all test our applications functionality, but rarely do we load and performance test these sites. Testing is so damn expensive. The fact is, more often then not, we don’t have a clue IF our NEW sales organization (our web sites) will actually  show up to work today. Where is the business intelligence in web deployment?

The answer could be found in Cloud Testing. First, we definitely need a new, affordable, and scalable test model that moves web performance analysis away from a simple math exercise. In the past,  if we had done any load testing (cost) we simulated only a fraction of the expected traffic to a site  (5%-10%).   Then, we would calculate what the impact of the other 95% of load would have on the site and application. A very inexact science, that has led to some pretty big and damaging headlines in the Wall Street Journal as many sites have crashed under stress and load. In another words, we need an affordable and scalable test model that allows us to regularly test 100% or 200% of our potential or expected load. Next, we need business intelligence tools specifically built to analyze the data that these tests produce.  Performance testing and monitoring produce huge amounts of data that needs to be analyzed to the tiniest detail. Finding the sources of latency and stress in applications and networks can be incredibly difficult if you have the wrong tools that only provide a single dimensional view of your results such as log files or excel spreadsheets. There is simply too much test data to review. What you really need are sophisticated BI tools that provide  an aggregated and correlated multi-dimensional view of the data in order to pinpoint and understand potential trouble spots. Given that our web sites are our new sales and marketing channels, web site deployment is an amazing area of uncertainty for all of us. We need a new testing model that delivers performance-based business intelligence before our sites go LIVE. We need CloudTest.

contact me at: tlounibos@soasta.com; twitter.com/lounibos

Email Us!
Subscribe to our Feed!
Join us on LinkedIn
Find us on Facebook
Follow our Tweets
See our pics