Performance Matters

The Effects of Using Think Time to Adjust Level of Load

Posted by Ben Polovick on Dec 13, 2011. 3 Comments

Some load tests are run with restricted resources, either because of deficiency in load generation muscle or from licensing constraints with the load-testing tool being used. Sometimes the performance engineer wants to quickly start a test and ramp up to peak traffic (however that may be defined) very quickly. In these situations, it is tempting to use the think times in your test case scenarios as a variable for adjusting the “level of load” that you drive. This “level of load” may be defined by “number of HTTP requests per second”.

To see where I’m going with this, suppose you have set up your test cases and workload to generate your target load with 500 virtual users and 15 second think times between pages. Now, suppose you need to generate the same target load, but with only 100 users. It might be tempting to assume that you can divide the think times by 5 and thus increase the throughput by a factor of 5, thereby allowing you to divide the number of virtual users by 5. The assumption can be stated as follows:

Assumption: If 500 virtual users with 15 sec think times yields x requests/sec, then 100 virtual users with 3 sec think times should also yield x requests/sec.

This assumption is flawed for a couple of big reasons:

  1. The response times are not being taken into account – both the think times and response times are factors in determining how much throughput your virtual users will get.
  2. This assumption requires making another assumption – your application will respond linearly with additional load, regardless of think time. That is, you are assuming that response times will be the same between the 500-user and 100-user workload. Additionally this assumes the application only cares about the average request rate and nothing else.

Point #1 above can be illustrated with basic math. A calculation of “average HTTP requests per second” can be made with the following equation:

We have two workloads in question. For now, let’s just go with the assumption that average response times should be the same for each workload. Let’s say we are dealing with 2.5 sec average response times and that each user is making 10 requests per page, on average (the HTML document + 9 page assets). Then we have the following:

500-user workload with 15 sec think times

100-user workload with 3 sec think times

Clearly the two workloads above are not equivalent in terms of requests/sec being generated. Now that we are taking the response times into account (and assuming they will be the same in each workload), we can set x = think time/page and then solve the resulting equation. Indeed, if you do this you will find that you need to use about a 0.1 second think time for the 100-user workload to generate the same throughput as the 500-user workload. How unnatural is it to have users with 0.1 sec think times? At the very least, it’s significantly different user behavior from the original workload.

The second point above can be illustrated with a real-world experience of mine before I had the benefit of leveraging CloudTest Pro’s ability to scale to enormous numbers of users. I was working with a tool that is licensed by the number of concurrent virtual users in use. There were periods of time when multiple teams within the company were simultaneously making use of that license, thus constraining the number of users that each team could use.

I was testing a back office application, which essentially had 2 different app layers and a database layer. The test case was simple. It merely consisted of 2 web service calls – one to get a sales tax calculation and one to do an order placement, with think time in between. The sales tax request was made directly to the 1st app layer, which made several 3rd party calls outside, then forwarded the request to the 2nd app layer, which managed connections to the database. The 2nd app layer would then make a JDBC connection to the database and read the corresponding sales tax data. My standard workload consisted of 800 virtual users with a think time modeled after data from production logs. Through a little bit of calibration testing I figured out that I could reduce the number of virtual users to 160, reduce the think times significantly, and have a workload that still generated the same number of sales tax and order placement requests per second.

After multiple rounds of testing, with both workloads, I had a discrepancy. Both the 800-user and the 160-user workload generated the same transaction rate against the application, but only the former hit a bottleneck in the app. With the 160-user workload, the application ran flawlessly. With the 800-user workload the database CPU pegged at 100% and the connection pool became completely saturated. It turns out that longer think times can consume more resources than expected. In this case, the longer think times caused the apps to maintain and manage more concurrent connections, specifically database connections. With longer think times, enough concurrent connections were created to max out the database connection pool, thus causing a bottleneck and forcing the queuing of some incoming requests.

The underlying theme here is that there are so many variables involved in any web architecture that it’s best model your virtual users as closely as possible to real-world users – model your think times carefully. In my case, the JBoss connection pooling configuration and connection timeout values were the unseen variables that responded to changing think times. Who knows what it will be in your case?

Stop Cheating on your Tests!

Posted by Dave Murphy on Oct 06, 2011. 2 Comments

I suppose we could have used a less inflammatory title for our recent webinar. It makes it sound like testers have been purposely doing something wrong. Perhaps we could have titled the webinar “Now you can execute more accurate and informative tests!” But the folks in marketing were right, and the intriguing title attracted our largest group of attendees ever. For those of you who didn’t attend, or if you did and would like to review the messages, you can watch the webinar here. This was the first in SOASTA’s latest webinar series, “Cloud Testing – Rewriting the Rules of Performance Testing”. Future webinars include “Run More Tests and Find More Issues” on October 27th and “Test On Your Schedule across the Lifecycle” on November 15th.

In this webinar, Scott Barber, President and CTO of PerfTestPlus, joins SOASTA’s VP of Performance Engineering, Rob Holcomb to discuss what performance engineers have done in the past to measure performance and find and fix issues; and why some of those techniques no longer reflect best practices. The focus is on web and mobile testing and why the higher scale, more distributed and often complex nature of that traffic is not well served by traditional testing tools or techniques.

After an introduction by SOASTA’s Brad Johnson, Scott, in his inimitable style, speaks to great effect about the four most common ‘cheats’ that performance testers have leveraged to overcome the constraints of inflexible test hardware, poor tool scalability, expensive pricing models and the lack of real-time information while testing. Scott begins his presentation by talking about the practice of modifying think times, typically to overcome licensing and/or hardware limitations imposed by the high cost of traditional load testing. His primary assertion: the only way to simulate production…is to simulate production. Interestingly, during the webinar a question came up suggesting that we’re testing computers, not humans, so why is accurately simulating user activity so important. In response, it was noted that it has become clear that variance in use absolutely has an impact on what happens to the infrastructure.

The second point discussed is the common practice of extrapolating results from a staging environment to predict what will happen in production. Architectures can be complicated, and the impact of those differences along with the additional complexities of ‘the real world’ make extrapolation problematic, at best. The best way to validate that your production environment will handle expected load is to test in production as part of your overall test strategy. (For more on testing in production check out this SOASTA webinar). Modeling user flows incorrectly is the third point addressed by Scott, reinforcing the notion that we’re not functionally testing the application, but need to make sure we’re putting a realistic load on the back end.

Finally, Scott presents a very interesting problem to illustrate the challenges associated with measuring performance, and how it can be as much an art as a science. Rob follows Scott’s presentation and, using SOASTA’s CloudTest, illustrates how we can use modern tools to, well, stop cheating. We hope you enjoy(ed) the webinar.

Implementing 2-Legged OAuth in Javascript (and CloudTest)

Posted by Dan Bartow on Jul 28, 2011. 2 Comments

Introduction

If you’re reading this you are probably looking for information on how to implement 2-Legged OAuth in Javascript.  I recently had to implement 2-legged OAuth into a CloudTest performance test for one of our customers.  Because 2-legged OAuth is not part of the official OAuth spec yet (as of 6/15/2011) there is relatively little information outthere about how to make this all work. Where there is information unfortunately it doesn’t universally work for all implementations since there isn’t a specification for it.  I hope this saves you some time… it definitely would have helped me out.  You will need a working knowledge of Javascript to find the implementation details in this article useful.  Without an understanding of Javascript you may find just the general OAuth overview interesting.

High Level OAuth Overview

OAuth is a way for applications to authenticate with one-another.  In essence a client application encrypts a string of values and passes that encrypted string, along with the values it used to encrypt it (except one, your secret key), to theserver.  The server then uses the values you sent across to look up your secret key and attempt to generate the sameencrypted string you did. The server then compares the two encrypted stringstogether.  If they match, it’s a success.  If not, it’s a failure.

The difference between 3-Legged OAuth and 2-Legged OAuth is that in the 3-Legged variant, the client first passes some credentials to the server and gets an access token back if authentication is successful.  Then this token is passed along in subsequent requests.  This is commonly called ‘the dance’ in OAuth developer circles.  When you authenticate with Netflix through various platforms (AppleTV, iPhone, Netflix.com), you do a 3-Legged OAuth dance.  This allows for users, applications, and authentication to be abstracted out into separate tiers.

Some other types of applications may be better suited for the authentication and message passing to happen in 1 request and 1 requestonly. This is where 2-legged comes in. In 2-legged OAuth you pass the encrypted string, the values used forencryption, and the message payload in 1 GET or POST.  If it is rejected, the message fails.  If it’s accepted then the message is processed.  This particular app that I was working on testing was a central logging system.  Every message was a log event.  There was no time (or functional need) for a three-way handshake in this app and also no notion of a maintained state.  2-Legged OAuth cuts out the middleman.  If authentication is successful the message is processed, no dancing around.

Click here to dive deeper

The Fragility of Web Applications

Posted by Dan Bartow on Jun 28, 2011. 2 Comments

Most web applications have some code in them that if utilized even slightly more than expected could send the whole stack toppling over. Web apps and their associated infrastructure are fragile and they must be performance tested thoroughly across all key functional areas. This testing should be done above expected traffic levels… the following is an example of why this is so important.

I was recently testing for a very large online retailer. Their site has the typical shop, buy, and self service functional areas. On this particular application there is an Ajax call to create an empty shopping cart (sometimes referred to as a transient session) as soon as you start browsing the catalog. It’s a lightweight and seemingly harmless GET request that passes through to the app tier to initialize the empty shopping cart. This cart is an in-memory object at the application tier.

What we discovered through testing was that if we generated the exact profile of traffic they were expecting with people browsing the catalog and creating empty shopping carts, along with customers adding products to the cart, then there was enough capacity to perform well. However, if we adjusted the load mix just slightly to have either more empty carts, or more carts with items in them, then the entire application slowed down and ultimately fell completely over. This affected not only people in the shopping experience but everywhere… the entire site went down.

This really got me thinking about how fragile apps really are unless you test all of the different components past their expected load levels, and assess not only their performance, but also the performance of the components around them.

Every web application has a weak link somewhere. Do you know where yours is? I bet that a very small load test that makes one particular type of request directed at your application could have catastrophic results. It could be 10% more users logging in than normal, or the worst case, more people trying to check out than you had planed for. I’m amazed at how many people market a flash sale and don’t change anything on their application to account for a totally different load profile than normal. We need to find those weak links and build them out to be more resilient.

Web Performance Optimization – Fun for Performance Engineers!

Posted by Fred Beringer on Jun 16, 2011. Leave a Comment

The Velocity conference is underway (SOASTA is one of the sponsor). Looking at the schedule, you will realize that topics cover all critical factors to take into consideration when building scalable, fast and reliable websites and services.

  • Operation: Deploying large and scalable cloud infrastructure, infrastructure automation, real-time monitoring, distributed systems, NoSQL databases etc.
  • Mobile performance: Analyzing mobile performance, optimization, writing fast client-side code, device comparison etc.
  • Web Performance: Optimizing server-side scripting with NodeJS, Client side optimization (image, javascript, browser specific optimization, HTML5 etc.), automated web performance testing etc.

Today’s performance engineers have the opportunity to get involved in all these area and this is why this is such a great time to be in performance engineering.

Want proof? Take a look at the following project some of our performance engineers tackled some months ago.

The web application to be tested was developed by the government of one of the largest country in the world. It was a form based application all adult citizens in the country had to use. We’re talking a massive number of users! The application was globally distributed across multiple private datacenters to offer failover option as well as provide the best user experience possible. The technical environment was your typical Struts application framework with an Oracle database for back-end. The target was to reach a clean test at 172,000 concurrent users using the application on the production environment. Yes, you read it right. On a production environment! Our performance engineers run a fair portion of their tests on production systems, especially at the end of the application development life-cycle. This is the environment the application is going to be accessed from and problems you find at that stage can’t be found when the application is installed in a lab or in a staging environment (Loadbalancer problem, CDNs, Bandwidth etc.). If you want to learn more about performance testing in production, there is a webinar available.

This was a 2 months projects and our engineers had some time to perform state of the art performance testing: Starting with a low number of concurrent users (500), fixing issues at this low level before moving on to greater level ie. 1000, 5000, 10k, 50k etc. That’s a fairly typical approach but sometime overlooked by some of our customers. We usually educate them while going through our performance testing methodology.

Many problems were found and fixed before reaching 100k: Servers misconfiguration, oversized pages, poor client-side caching, SQL optimization, login process optimization (One of the typical optimization engineers have to deal with regularly as companies tend to retrieve all kind of heavy information during the login procedure).

One problem really drove our engineers crazy for a few weeks. (The type of problems they love as it is really challenging and rewarding to get them fixed!). At a fairly substantial level of concurrent users (110k+), they were observing the following real-time chart:

They ran hundreds of tests at this level and were getting plagued by the same issue: As soon as the first users were done filling up their form and submitting it, overall response time was getting to the roof, throughput dropped significantly and error rate reached 6.5%! Not a pretty picture and a really bad user experience: Spending 15 minutes to fill governmental forms and getting an error when submitting is not what most users want to go through.

After many days scratching their heads, the problem was finally identified. This application was distributed across the world with a global Load Balancer taking the requests and route it to local load balancers closer to the user. As it was a highly secured application, the global load balancer was responsible to serve a highly encrypted (2048 bits) certificate during the submit process. When it was serving this certificate, CPU on the load balancer would go sky high and the throughput would drop. Our engineers had the idea of reducing the encryption level to 128 bits as a test and sure enough the CPU level on the load balancer was normal and it was being able to serve the certificate as expected. But as soon as the encryption level reached 2048 bits, problems were back. We ended up contacting the load balancer manufacturer and with all the data collected during tests it was all the information they needed to provide a firmware fix.

This is the final test results with the firmware fix.

Very clean test! Very low average response time (440ms), extremely low error rate (0.001%!) and 172,000 virtual users getting through the process without a itch!

That’s what performance engineering is all about these days! Getting involved with large scale projects, learning the full scope of performance optimization, teaching customers performance best practices. And best of all, there is product innovation to bring engineers all this fun! CLOUDTEST! And soon a very big surprise for all performance engineers in the world … !

Now is the Time to be in Performance Engineering!

Posted by Fred Beringer on Jun 09, 2011. 2 Comments

I’ve always considered performance engineering as the most rewarding discipline in software testing. In my opinion, this is where you have the most opportunity to learn, especially technically. Great performance engineers follow Cem Kaner principles described in his Bug Advocacy paper and especially this one:

The best tester isn’t the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.

It’s about finding the right ways to communicate problems and giving as much useful information to the developers, DBA and IT guys responsible for the infrastructure where the application under test resides. It’s about dealing with objections from these people, motivating them to consider the problem seriously and to start investigating it. It’s also about pinpointing the problem in the right direction. Great Performance Engineers need to be good salesmen and need an amazing amount of knowledge to get the issue they’ve found fixed, whether it’s in the application code, the infrastructure in which the application resides or elsewhere in the overall architecture!

Great Performance Engineers get to learn about:

  • The intricacies of load balancers, especially since they’re one of the primary sources of contention when dealing with high volume applications. A lot of companies take load balancer configuration for granted and don’t bother testing their algorithm under load.  A BIG mistake!
  • CDN configuration. Again one of the top problems our Performance Engineers find when testing applications from outside the firewall.
  • Bandwidth usage and its implication on the overall performance of the application.
  • Auto-scaling mechanisms.
  • Garbage collection, memory leaks, unoptimized database schema and queries, optimizing CPU consumption, etc.
  • Everything about front-end optimization: Browser caching, expired headers, cache busters, image optimization, lazy loading, progressive rendering, etc.

Performance Engineers are able to test today at a scale they couldn’t dream about 4 years ago. Look at the test below: a 58 min test with 7 Terabytes of data received! A “big data” problem Performance Engineers can have fun with these days.

They’re able to test from inside and outside the firewall, providing coverage for problems they couldn’t previously replicate. They can, with CloudTest, get performance results in real-time and have conversations with developers, DBAs, Ops and other IT constituents during the test, increasing their chance to solve problems quickly, and to learn. A recent engagement with a large telecommunications company in the US brought 90 people together during the 2 hour test. A great learning opportunity!

If you’re eager to learn, and help companies get the best performance from their application, this is the best time to be in performance engineering. Best of all, SOASTA is hiring!

At SOASTA we’re in the wisdom business

Posted by Fred Beringer on Apr 07, 2011. 2 Comments

An interesting discussion with a former LoadRunner Product Manager piqued my interest and triggered this post. The real trigger was this part: CloudTest (like others, including LoadRunner) is still just a testing solution and as such serves as a filter or gate-keeping function in the lifecycle. [attrib: Jim Duggan, Gartner]. I’m gonna disagree on that one as I’ve had a different opinion on the “gate” part for the past 15 years. In my opinion, the goal of performance testing (and software testing in general) is not to be used as a gate. Testing is only one part of the equation when you need to decide whether or not to go live. There are other important factors to take into consideration, and depending on the context and timing they might be more important. The marketing and sales side of the software industry often prevails, for better of for worse.

The true goal of Performance Testing is to provide wisdom. Maybe the sole purpose of LoadRunner is to be a testing solution that serves as a gate, but at SOASTA we’re in the wisdom business.


The first objective of performance testing is to gather relevant data from EVERY components involved in the overall performance of your application. This is your raw and dumb data.

  • Data from your application itself, memory, CPU consumptions, number of processes, heapsize, etc.
  • Data from the application’s underlying infrastructure: Application servers, web servers, databases, SSL servers, CMS, memcache servers etc.
  • Data from your application’s ecosystem: CDNs, Load balancers, switches, routers, DNS, etc.

You end up with LOTS of data. To give you an idea of the magnitude of data you might gather, one of the tests we’ve performed with a major TV network generated over 7 terabytes of data in less than one hour. Transfer was 17GigaBit per second! Thanks to CloudTest we’re able to gather these terabytes of data in real-time. But they’re USELESS data if you don’t have a mechanism to transform them into information. In order to deal with this BIG DATA PROBLEM you need a real-time, in memory OLAP engine. Gartner predicts that By 2014, 30 percent of analytic applications will use in-memory functions to add scale and computational speed. It looks like SOASTA is ahead of the curve!

Information is created by analyzing relationships and connections between data. Information should help you make some sense from your data and become relevant to your business:

  • What’s the relationship between the number of virtual users and the memory consumed by my application server?
  • How much server capacity am I left with when I reach 10,000 concurrent users during my test?
  • What is the correlation between the number of process counts on the database server and the overall throughput?
  • At what stage during the test do I see a drop in overall response time? Can I correlate this drop with another data to understand this behavior?
  • What is the correlation between an increase in response time and the number of errors coming from my SSL Server?
  • Why is 90% of response time for my overall homepage taken by this particular file? Where does it come from? Why does it take longer than the other page assets?

By combining, correlating and aggregating your data you’re able to build enough information to understand the behavior of your application and its entire ecosystem.

Knowledge is created by receiving, absorbing and understanding the information. From this knowledge we can make decisions and take actions. If you’re observing a surge of traffic going into one particular webserver, impacting the overall response time for some of your visitors, you can pinpoint the problem to a misconfiguration in one of your load balancer. You’ve got sufficient knowledge to take action and make a change on the fly. That’s what we call actionable intelligence at SOASTA. And with modern testing tools, such as CloudTest, it can happen in real-time. That’s the agile way of doing performance testing!

Wisdom deals with the future and predictability.  This when you apply knowledge to change your process and reach your true objectives. This is the level where you test your predictions, execute them, monitor the results and adapt. As an example, what if you’re expecting 200,000 concurrent users on your website after a big product announcement? You think your visitors will behave in a particular way (you’ve got historical data to back this up): 50% browsing the site, 15% login, 15% putting items in and out of the shopping cart and 20% watching a video. How much bandwidth do you need? What should be the right configuration for your load balancer? What assets should be within the CDN? What should be the configuration of the memcache server? What if one of the web server crash? What would be the impact for your visitors? What would be the impact on your business?

Today, a lot of companies stop at the information stage and analyze their metrics retroactively to understand what has happened. There are some behavioral reasons, but most of them are just not equipped to reach further levels. You need a mechanism to:

  • Gather raw data in an efficient manner.
  • Build information by combining, correlating and aggregating these data to start to understand behavior.
  • Bring you knowledge, in real-time, so you can make decisions and take actions, FAST.
  • Perform predictive analysis, build prediction models so you can help your business and your customers in the best possible way. Gartner predics that by 2014, 30 percent of analytic applications will use proactive, predictive and forecasting capabilities. SOASTA is already there!

At SOASTA we’re not in the gate business. We’re in the wisdom business.

Cover Your Bases When Performance Testing

Posted by Dave Murphy on Mar 31, 2011. Leave a Comment

With the release of CloudTest Pro, SOASTA has reinforced a central theme of our Performance Test Methodology: expand your scope to include tried and true traditional methods of testing with the benefits of cloud testing: scale, speed, accuracy and flexibility.  CloudTest Pro makes it much easier to execute both internal and external testing without sacrificing control over your testing processes.  Also, it’s not just a matter of where you generate the load, you must test with a purpose.  CloudTest Pro CloudTest Pro makes it much easier to execute both internal and external testing.  We believe it’s not enough to simply measure end user experience or find out when bandwidth is saturated.  Important information?  Of course, but hardly the only keys to confidence.

To gain confidence you need to identify the things that are in your control and that you can address.  Can you improve the performance of the ecommerce shopper who’s at the end of a DSL line on a 3-year-old computer while the Roku box is streaming Netflix?  Of course, but not by buying all of your end users faster Internet access and better computers.  It’ll be by doing the same things you’d do to improve performance for everyone: ensuring that your web pages, applications, infrastructure, network and third party providers are performing optimally.

The main tenants of the SOASTA’s Performance Test Methodology include:

  • Testing both in the lab and live, web-based applications in production
  • Leveraging the Cloud to test at both typical and peak traffic levels, from hundreds of users to millions
  • Responding to the acceleration of development cycle times by making agile performance testing a realistic alternative
  • Generating geographically dispersed load for a valid representation of real-world traffic
  • Generating both internal and external load and using both the lab and production environments for the most efficient and effective results
  • Analyzing Performance Intelligence in real time to speed problem resolution

We’ve advocated external testing because it’s important to understand the performance from different geographic regions.  It’s also the most realistic and valid way to generate user traffic.  A complete suite of testing includes tests designed to identify specific issues or establish baseline metrics.  Comprehensive testing will often include stress, spike, endurance, failover, capacity and break-point tests in addition to understanding response times.  Make sure you’re getting a 360 degree view of performance.  For more thoughts on the topic, check out this post from last year: What Does Performance Testing Mean? http://www.soasta.com/2010/05/06/what-does-performance-testing-mean/

Best Practice Architecture – The Graceful Turn-Away Page

Posted by Dan Bartow on Mar 17, 2011. Leave a Comment

One thing that I’ve seen best-in-class web applications make use of is the ‘graceful turn-away page’, also known as the graceful deferral, turn-away, or throttle page.  I apologize in advance for this post being a little light on technical content, but the reason for that is a throttle can be implemented in a number of different ways depending on application architecture.  Therefore I’d like to just focus on why this should be a mandatory piece of functionality for any online application.  I will talk a little bit about how this is done in some applications and the rest is up to you, the smart people, to decide on how to implement it for your application.

First things first – what is it?  Simply put, it’s application functionality that throttles new users and requests flooding into an application while preserving the experience for people already on the site.  Without queuing and throttling mechanisms applications that get more traffic than expected will normally blow up and impact everyone.  Throttling will give new users coming in a graceful page saying something to the effect of ‘we are experiencing higher traffic that normal, please come back later’.  It stops people at the front door of the store while letting everyone else inside the store finish shopping.  The key here is that sessions already established and in flight are not impacted at all.

One of the best examples I’ve seen of this is for event registrations held by Active Network.  Active Network does registration for things like marathons, triathlons and Iron Man events.  These events typically have a fixed registration count but get many, many magnitudes higher users coming to the site trying to get one of those spaces.  To further complicate matters, registration opens at a very well-publicized, set date and time.  So the traffic profile is something like this: they have potentially tens of thousands of people sitting on the web site page clicking refresh while waiting for the event to open.  At some point they flip a switch and let everyone in.  First come first, serve on the seats available.  At some point though they may have such an unexpected spike in traffic that they need to queue people at the front door and let people in the registration process get through with a good customer experience.  If a spike like this happens, a nice friendly turn away page is shown to all new users.  Performance is great for everyone still in the registration process.  Brilliant.

I have seen this on some best in class web apps, but not nearly enough.  I feel that this functionality should be mandatory for every site out there.  Again, not just a friendly ‘fail page’ that everyone sees when things blow up.  This should happen way before that.  Any web server can show a branded error page.  They key here is the throttling capability.  There is usually some effort to make this work properly but it’s well worth it.

In order for this functionality to work properly it is usually implemented at either the application tier, the load balancer tier, or some combination of the two.  The application server usually knows how many sessions are present in the application.  Application servers like WebSphere and JBoss know how many in-memory sessions are present and can report this out over JMX.  This is a great metric from which to base concurrent load levels.  Web or application server threads or connections are another common metric to watch for making a decision on when to turn away.  The load balancer is a great place to do the actual throttling because most modern load balancers, like F5 Networks BigIP boxes, can serve static html pages.  Plus they are aware of how many connections are open to the web and/or application servers.

Does your web app have a graceful turn away or throttling mechanism to handle unexpected traffic?  If not, I would highly recommend such functionality.  Create it and test it to see that it works as intended.  It doesn’t even have to be automatic. It’s okay to go out on a load balancer and manually put the page up if that’s what it takes.  As long as it holds back the flood of traffic from the tiers that  need to be running optimally to serve the existing customers then it’s a win.  It will create a better customer experience for everyone during turbulent traffic periods.  If you can’t keep the site from going down entirely, at least you won’t have a zero dollar revenue window because the whole thing blew up.

The Significance of SOASTA CloudTest Grid

Posted by Dave Murphy on Mar 02, 2011. Leave a Comment

With the release of CloudTest Pro, SOASTA’s customers now have direct access to one of my favorite capabilities as a SOASTA performance engineer: the CloudTest Grid. The CloudTest Grid continues our success in reducing the cost, complexity and time to results for performance testing websites and applications.

Like any other engineer, a performance engineer needs timely access to professional tools before they can add real tangible value; i.e. delivering evidence as to whether or not a website or application will meet target responsiveness and scalability requirements, and help identify bottlenecks and their root cause. Not only does the engineer need access to professional tools, the tools need to be suitable for the job.

Most performance testing tools require significant capital investment in licensed software and hardware infrastructure, as well as ongoing maintenance costs just to test what is typically a scaled-down version of a production environment. This may have been appropriate for the internally facing applications of the past where user numbers have been counted in the tens or hundreds. However today’s web applications are delivering the next generation of dynamic content to an extraordinary number of global users every day. Keeping pace with the speed and agility of today’s web development initiatives requires a new generation of testing and performance management solutions.

Before joining SOASTA I found that it was becoming increasingly difficult to help organizations justify investing in traditional performance testing tools simply due to the cost and complexity of these offerings. Often, by the time a business case had been constructed and a flexible commercial model agreed upon the initial requirement for testing had passed.

These challenges have led to cloud testing as a way to reduce costs and provide the benefits of scale and more accurate testing. However, there remained the challenge of how to best leverage the distributed nature of the cloud and optimize deployment to be as efficient as possible.

With CloudTest Grid, performance engineers can deploy hundreds, or even thousands of load servers around the globe in a matter of minutes. The servers are automatically spread across locations and cloud providers to meet the requirements of each specific test, and the Grid takes care of all error checking and fail-over to replace any bad instances or get additional capacity from other locations, as needed. Once deployed, the servers are monitored throughout testing to track their health and ensure the Grid is operating as expected.

With SOASTA CloudTest Grid it is possible to provision a complete test environment, when required, and just for the period of time that is needed, which significantly reduces the time to test. We routinely perform tests involving tens of thousands or hundreds of thousands of simulated real users. Combined with our industry leading performance analytics SOASTA CloudTest delivers increased business value over traditional testing tools at a fraction of the cost with significantly improved time to value.

Email Us!
Subscribe to our Feed!
Join us on LinkedIn
Find us on Facebook
Follow our Tweets
See our pics