If you ask someone in the software engineering field what performance testing means, you will likely get a variety of answers. “It’s testing to make sure that when users hit pages or make requests, they come back fast”. “It’s making sure that a site can sustain peak traffic levels”. “Making sure that my application doesn’t have any memory leaks”. All of these are valid statements, and each of them is an important part of a comprehensive test plan.
There is a problem here: Companies are either doing 1 or 2 of the important things and neglecting the rest, or not testing at all because of plain old confusion. I’ve see a lot of companies that can tell you the response time of every asset in a page, but can’t tell you if they can support their peak anticipated traffic levels. I’ve seen companies that know they can support 50,000 concurrent users but have no idea what the page response time will be at that level.
Here are the main categories I see in the industry with respect to performance testing:
- Capacity – How many users can I support?
- Experience – How fast is a page responding?
- Reliability – Does my site stay up? How does my site react to failures?
- Scalability – Can my platform scale to meet growth?
- Transaction – How long does it take to go through n number of pages?
- Code level – How fast are individual pieces of code? Is there contention?
As you can see, all of these are important. However, I rarely see engineering teams doing more than 1 of the types of testing listed above. Why is that? It’s primarily because there is a company associated with each of those types of testing and an associated product to go along with it… no one is evangelizing the entire picture.
In the following diagram, I present a view of performance that I believe represents the priority of performance testing objectives from the customer’s perspective.
First and foremost – is your site going to crash under peak or unanticipated load levels? If so, this is a huge risk. A site can be all text with one image, but if it completely bottlenecks under peak load then it doesn’t matter if it’s fast. If you’re expecting 2,000 users or 200,000 users at peak, the whole architecture has to support that first and foremost. There have been studies published that say customers would rather have a site be consistently slow but up all time, than have it be fast with inconveniencing crashes. Makes sense to me.
Okay – now, if you know your site can fly during peak load levels (during something like a Super Bowl commercial, after an email campaign goes out, or if you get linked on Slashdot, etc), now the team can focus on optimizing page speed. We all want our favorite sites to be fast, or faster in a lot of cases. But again, it has to be up all of the time first. Not just for your company’s image, but so you don’t lose revenue when you have the most opportunity to gain it. This type of testing is really important, but let me state my belief again: If you are performance testing to optimize page and transaction speed, but don’t know what your capacity thresholds are… I would stop immediately and think about capacity testing first.
Capacity? Check. Fast pages? Check. What happens when a web server drops off the grid? A database node? How is the performance after the application has been running at peak for 10 hours? Failure testing and duration testing now come into play.
Now, what if you need to add capacity above and beyond expected load levels? You need a scalable architecture that is tested and proven to add performant capacity to the architecture.
Transaction and Code Level
Focusing on small amount of code and testing isolated functionality under load would be like a tire company testing the tire in isolation with no regard for the how the car fits into the picture. Start from the customer and test inwards towards the center of your application. Literally and strategically. If the performance lab is only testing isolated pieces of code or functionality under load, there are a lot of important questions going unanswered.
Each of these categories also has an impact on how you monitor your application. The things you need to monitor vary greatly if you’re looking at speed of pages vs. capacity thresholds. Ill talk about monitoring in a future post.
At the end of the day, is what you’re testing and how you are testing it helping you achieve your business or engineering objectives? If so, and you’re only responsible for one piece of the puzzle I laid out, then is someone responsible for the other pieces? If not, there is an opportunity to test more comprehensively for greater confidence in the overall application performance, scalability, and reliability.
About the Author