At SOASTA we’re in the wisdom business
An interesting discussion with a former LoadRunner Product Manager piqued my interest and triggered this post. The real trigger was this part: CloudTest (like others, including LoadRunner) is still just a testing solution and as such serves as a filter or gate-keeping function in the lifecycle. [attrib: Jim Duggan, Gartner]. I’m gonna disagree on that one as I’ve had a different opinion on the “gate” part for the past 15 years. In my opinion, the goal of performance testing (and software testing in general) is not to be used as a gate. Testing is only one part of the equation when you need to decide whether or not to go live. There are other important factors to take into consideration, and depending on the context and timing they might be more important. The marketing and sales side of the software industry often prevails, for better of for worse.
The true goal of Performance Testing is to provide wisdom. Maybe the sole purpose of LoadRunner is to be a testing solution that serves as a gate, but at SOASTA we’re in the wisdom business.
The first objective of performance testing is to gather relevant data from EVERY components involved in the overall performance of your application. This is your raw and dumb data.
- Data from your application itself, memory, CPU consumptions, number of processes, heapsize, etc.
- Data from the application’s underlying infrastructure: Application servers, web servers, databases, SSL servers, CMS, memcache servers etc.
- Data from your application’s ecosystem: CDNs, Load balancers, switches, routers, DNS, etc.
You end up with LOTS of data. To give you an idea of the magnitude of data you might gather, one of the tests we’ve performed with a major TV network generated over 7 terabytes of data in less than one hour. Transfer was 17GigaBit per second! Thanks to CloudTest we’re able to gather these terabytes of data in real-time. But they’re USELESS data if you don’t have a mechanism to transform them into information. In order to deal with this BIG DATA PROBLEM you need a real-time, in memory OLAP engine. Gartner predicts that By 2014, 30 percent of analytic applications will use in-memory functions to add scale and computational speed. It looks like SOASTA is ahead of the curve!
Information is created by analyzing relationships and connections between data. Information should help you make some sense from your data and become relevant to your business:
- What’s the relationship between the number of virtual users and the memory consumed by my application server?
- How much server capacity am I left with when I reach 10,000 concurrent users during my test?
- What is the correlation between the number of process counts on the database server and the overall throughput?
- At what stage during the test do I see a drop in overall response time? Can I correlate this drop with another data to understand this behavior?
- What is the correlation between an increase in response time and the number of errors coming from my SSL Server?
- Why is 90% of response time for my overall homepage taken by this particular file? Where does it come from? Why does it take longer than the other page assets?
By combining, correlating and aggregating your data you’re able to build enough information to understand the behavior of your application and its entire ecosystem.
Knowledge is created by receiving, absorbing and understanding the information. From this knowledge we can make decisions and take actions. If you’re observing a surge of traffic going into one particular webserver, impacting the overall response time for some of your visitors, you can pinpoint the problem to a misconfiguration in one of your load balancer. You’ve got sufficient knowledge to take action and make a change on the fly. That’s what we call actionable intelligence at SOASTA. And with modern testing tools, such as CloudTest, it can happen in real-time. That’s the agile way of doing performance testing!
Wisdom deals with the future and predictability. This when you apply knowledge to change your process and reach your true objectives. This is the level where you test your predictions, execute them, monitor the results and adapt. As an example, what if you’re expecting 200,000 concurrent users on your website after a big product announcement? You think your visitors will behave in a particular way (you’ve got historical data to back this up): 50% browsing the site, 15% login, 15% putting items in and out of the shopping cart and 20% watching a video. How much bandwidth do you need? What should be the right configuration for your load balancer? What assets should be within the CDN? What should be the configuration of the memcache server? What if one of the web server crash? What would be the impact for your visitors? What would be the impact on your business?
Today, a lot of companies stop at the information stage and analyze their metrics retroactively to understand what has happened. There are some behavioral reasons, but most of them are just not equipped to reach further levels. You need a mechanism to:
- Gather raw data in an efficient manner.
- Build information by combining, correlating and aggregating these data to start to understand behavior.
- Bring you knowledge, in real-time, so you can make decisions and take actions, FAST.
- Perform predictive analysis, build prediction models so you can help your business and your customers in the best possible way. Gartner predics that by 2014, 30 percent of analytic applications will use proactive, predictive and forecasting capabilities. SOASTA is already there!
At SOASTA we’re not in the gate business. We’re in the wisdom business.
Fred – you have been developing your wisdom about performance engineering for many, many years now. How did you get to that level of expertise? What tools did you use before SOASTA existed? How did those tools help you to learn?
Good question Mark. For many years, I think it’s been very difficult for everyone involved in performance testing to reach a complete level of expertise. The tools we’ve been using for years, including Loadrunner, have not been able to cope with ever demanding applications, especially web and mobile applications. I truly think CloudTest is a dream come true for every performance engineer as they can reach a high (and complete) level of expertise much faster and more affordably than with any traditional tool, particularly Loadrunner.
Why was it so difficult to reach a complete level of expertise with traditional tools? 3 main factors in my opinion: Scale, speed and affordability.
(1) Scale
* It’s been impossible to generate realistic load (as observed on a production system) with traditional tools. It’s difficult to reach true expertise when you’re not able to test at scale. You’re left with extrapolation which is, in my opinion, black magic at best.
* Most traditional tools are fat client. Maybe fit for client/server kind of applications but unfit for web and mobile applications. Additionally, when you have to manage a global team, you want to give access to your performance tools to all your team in an easy way. No such luck with loadrunner for example. This was a huge limitation for the distributed organization I was managing prior to joining SOASTA.
* Most traditional tools limit themselves to testing behind a firewall. While this is a necessary step, it’s definitely not enough if you want to uncover all problems your application might (will) encounter while running on a production system. If you want to find problems with your CDNs, your load balancer, DNS, bandwidth etc. you need to test outside of your firewall. If you don’t test both internally and externally, your expertise will definitely be incomplete.
* If you want to reach true level of expertise, you also want to expose your application to realistic traffic. And this mean traffic coming from multiple geographies and not only from your own environment.
* You want to be able to collects and interpret terabytes of performance data in real-time, even when you test with millions of virtual users. As far as I know, Loadrunner becomes very sluggish with high volume and long test. This is a real limitation. You’re basically left with offline analysis. Not the best way to quickly improve your expertise.
(2) Speed
* Most traditional tools rely on proprietary scripting language with a high learning curve. You don’t want to learn a proprietary language to become an expert in performance testing. You want to leverage a language what you already know, such as javascript, and use an easy to learn browser UI.
* You want to automatically provision your load servers and not spend hours to manually provision your load agents. Provisioning is not what performance testing is all about. Performance engineers shouldn’t have to worry about it.
* Obviously if you want to become an expert in web performance testing, you want to use a tool which supports latest web technologies ie. AJAX, Flash, HTML5 and not a tool with incomplete support with these technologies.
* You want to be able to build-once and reuse your test, especially when dealing with dynamic data. Most traditional tools, including Loadrunner, use manual dynamic session or hard-coded rules. You end up with significant repetitive work and long support delays.
(3) Affordability
Cost can be a very limiting factor for someone who wants to reach true expertise. In my previous position I had invested in LoadRunner and it’s been a bad decision (My mistake I guess). I was left with:
* Prohibitive cost to use it in all my development centers.
* Prohibitive cost to get anything above the 10k VUs level (software only).
* Prohibitive cost to handle different protocols.
* Prohibitive cost to test externally. I was left with high rates for managed services.
* Significant hardware procurement, administrative and maintenance cost. This is 2011, I should’t have to worry about this. This is not what performance testing is all about.
* Low scale, high cost for single location.
* Recurring (crazy) high maintenance cost with questionable support and updates.
I want a tool which brings me unlimited internal usage, unlimited analytics, unlimited use of web protocol, unlimited internal and external VUs, low cost for my hardware environment dedicated to performance testing (public and private clouds). I want to test at any time, to any scale with everything included in my subscription. This tool is CloudTest Pro and it’s the best fit today for all performance engineers in the world who want to reach true expertise.