Here’s an honest question: How do you feel about load testing — honestly?
Now what if you could love testing as much as you love creating, building, and deploying applications? If the tests were relevant, scalable, and easily integrated into your building environment — how would that feel?
Imagine load testing that happens early and often. Testing that doesn’t stop the workflow but gives crucial information about bottlenecks in your application. Imagine building an application whose user experience is no less than a joy. Where the code is functional poetry and the thing also performs like a Bughatti Veyron on hyperdrive?
All the testing, all the time
Within the vast realm of modern application testing, you’ve got two broad sets:
- Functional (AKA acceptance) testing
- Performance testing
Functional testing measures how well your application works: that the code is doing the right stuff. (Unit testing checks that the code is doing those things well).
Load testing sits under the performance testing umbrella. Although there is quite a bit of wiggle room about where the lines defining each type of test should be drawn, you could make the argument that performance testing measures how well the user experiences your application, and load testing measures how well those experiences stand up against volume.
Why performance testing?
Performance testing used to be a nice extra. Sure, the application worked great, but could you make it faster? Could you make it perform better under stress? Could you ensure that it would work as intended on Black Friday (under heavy traffic loads) as well as at 2am on a regular Sunday?
WATCH THE VIDEO: Load Testing for Your Peak Season
When you look at it like that, it’s obvious that in today’s digital marketplace, performance is as essential as function. You can’t just have an application that works: you must know it works under pressure for real users.
Performance in the modern world happens under increasingly complex constraints. Your users aren’t just on desktops. They’re on smartphones and switching to desktop, tablet and IOT throughout the day. Your application needs to look good and perform on all of these devices, staying stable between sessions.
You’re building complex architectures shared by several applications using new technologies that change continuously. All of this improves your users’ experience — as long as everything performs as well or better than expected.
And those users are increasingly demanding: they (think about it: yourself included) expect applications to load quickly and perform seamlessly between devices.
Performance testing in a DevOps culture
DevOps sounds like something out of Mission: Impossible. And for those who have been in the tech community for awhile, the idea that developers and sysops people could work seamlessly together on complex projects with major time and budgetary constraints, “mission impossible” might seem like a perfect name for it.
DevOps prizes communication, collaboration and integration between dev and ops. It’s an essential part of continuous delivery, with automation being central to its success.
The benefits are huge:
- Improved deploy frequency which can lead to faster time to market
- Lower failure rate
- Shortened lead time
- Faster mean time to recovery
It isn’t hard to convince most people that DevOps is a good idea. Convincing them that it’s possible is another thing entirely.
Traditionally, the development team scored gold stars based on the number of features they could roll out, while the operations team was tasked with keeping things stable. This could, can, did, and does lead to conflict. In a modern agile DevOps workflow, however, one team works together to produce a stable, feature-rich application that performs under pressure. It’s happening and if you’re not on board yet, you will be soon.
Creating an ideal load testing strategy
When you load test, you are answering these questions:
- Can your application handle N users?
- Does your application meet your response time goals?
- After a load peak, how does your application behave?
- How many users can the application handle while maintaining your set response time goal?
- What is the load threshold above which the server(s) begins to generate errors and/or refuse connections?
- Does the server(s) remain functional under high load or does it crash?
It used to be that performance testing could only realistically happen after most of the application was built. In the last quarter of a project, testing would become meaningful. And that made sense. But it also led to an elite performance team being called in at the last minute, performing tests and finding unfixable, last-minute issues that often got brushed under the carpet or slapped with a band-aid solution. Not a great workflow.
With modern load testing solutions, combined with a continuous delivery/continuous integration application, you can load test early often and continuously.
But you still need a comprehensive strategy or your tests are close to worthless.
WATCH THE VIDEO: How to Do Continuous Load Testing with CloudTest and Jenkins
1. Start with risk assessment
At the 2004 Workshop on Performance and Reliability, Ross Collard said:
“We cannot test everything, and so we need to make conscious decisions about where to focus the depth and intensity of testing. Typically, the most fruitful 10% to 15% of test scenarios uncover 75% to 90% of significant problems.”
Knowing this, you need to prioritize and acknowledge these potential risk factors in your project:
- Resource-intensive features
- Time-sensitive uses
- Likely bottlenecks
- User impact/user experience
- Complex features
- Troublesome/poorly built/poorly maintained areas
2. Consider your performance goals
Your goals and objectives will be unique to your system, but consider the following objectives when creating your goals:
- Realistic response times
- Operates well when accessed simultaneously by multiple users
- Handles heavy loads
- Performs reliably over extended periods of time (soak tests)
- Uptime must be adequate
- Resources must be used effectively/efficiently
- When pushed beyond its planned maximum capacity, the system must degrade gracefully and not fail catastrophically
3. Create your load testing strategy
From the cross section of perceived risks and goals, create your load testing strategy. In the end, you should have a document with the following information:
- Answers to your risk assessment
- Modelling, including logic used
- Your buyer/user journey, compartmentalized into components
- Buyer/user journey with calculation of think times/business transaction
- Your service level agreement
- Description of each test
- Testing environments
These are the six points I’m hoping you take away from this post today:
- Consider risk assessment — What are the weaknesses in your project? Test for those.
- Create a strategy — Have a plan.
- Keep your user in mind.
- Function and performance matter.
- Test early, often and continuously.
- Be ready and able to report to all your stakeholders.
About the Author
Tom has more than 20 years of experience as a manager and product manager in the software development tools field. Today, Tom works in product management as Senior Evangelist at SOASTA, the leader in performance analytics. He speaks frequently at industry conferences and meetups on topics, including Web app performance and testing at large scale, mobile continuous integration and testing, automated mobile testing tools, and big data analytics for business value.