By now, the performance and availability issues that have plagued Pokémon Go aren’t news — they’re history. (For a quick history lesson, read this, this, and this.) And you know what they say about people who don’t learn from history. In this post, I’m going to share some lessons we should all have learned from Pokémon Go.*
At some point, most of us have earned battle scars from dealing with server overload. In a funny way, these scars are a badge of honour. They show that we’ve launched a product or campaign that quickly went even more viral than our greatest hope or reckoning. Were the folks at Niantic confident that Pokémon Go would be a success? Probably. Did they think it was going to smash records and become the biggest mobile game of the decade? Probably not.
Here are four ways Niantic could have prepared — from a functional and load-testing perspective — if they had known they were about to launch one of the most successful mobile apps of the last ten years.
Performance issue #1: Pokémon Go wouldn’t open
This is a classic “buckling under pressure” load problem. According to Niantic’s CEO, the game’s servers were (and still occasionally are, according to Down Detector) overloaded with just the three countries that it launched in.
End-to-end testing on their website, mobile app, and API backend would give Niantic a view of all their performance data while tests run — from back-end systems to front-end performance. They could test to the expected level of usage and beyond.
- Testing is more than functional. It has to include performance testing, both at the front end (what the user sees) and the back end (what users don’t see).
- Predicting load for a new release event can be difficult without past data. The solution is to test at expected load and then increase to 2x, 4x, or more. This will expose system weak points that will fail under increased load.
- Measure the functionality and performance of the mobile app when the back end is under load. This will reveal which parts of the app are affected when back-end services are slow or unresponsive.
- Use APM tools to monitor the health of the back-end applications and systems and correlate with load and front-end performance data.
Performance issue #2: Battery drain caused by the game
This is a known issue with Pokémon Go. If you’ve developed a robust app with a lot of functionality, here’s how to keep it from excessively draining users’ batteries:
- Measure app performance, tracking CPU usage, battery level, data I/O on real devices to understand how app usage impacts user experience (e.g. does the battery drain more quickly in this app and how might that be remedied).
- Compare baseline performance to the current build to catch battery usage changes.
- Test on every device using mobile device clouds so that developers and testers can access the latest devices.
- Integrate performance tests with Continuous Integration and Continuous Testing processes so tests can run simultaneously on multiple devices with the check-in of every build.
Performance issue #3: Game doesn’t have accurate location information
See the answer to #2.
Performance issue #4: GPS not found
By doing functionality testing, Niantics could have easily recreated GPS-related problems and figured out a better way to handle them than the generic “GPS signal not found” generic error message.
Takeaway: Being an industry leader means you’re on the front lines
And when it comes to tech, the front lines are where the hard battles are fought and the hard lessons are learned. Niantic has made a huge leap forward in the popularization of augmented gaming. We owe them a debt of gratitude — not just for bringing us an awesome new game — but for helping us learn from their experience.
*Disclosure: I’m proud to announce that, despite the app’s hiccups, I’ve made it to level 5. Woohoo!
- The complete guide to performance testing your websites and apps [EBOOK]
- The seven steps to pragmatic mobile testing [EBOOK]
- A 7-part guide to sanity check the hysteria behind holiday website outages
About the Author
Ann has spent over two decades leading some of the most disruptive innovations in software-as-a-service, cloud computing, cybersecurity, e-commerce, data analytics, mobile software, networking and embedded system technologies. She is a leading advocate for better customer experience management and believes today's mobile and hyper-connected consumers deserve web and mobile applications that are fast, reliable and easy to use. Prior to SOASTA, Ann was Senior Vice President and Chief Marketing Officer at LiveOps. Ann also served in a variety of management roles at Symantec, Sybase (acquired by SAP), eBay and HP. Ann received her bachelor’s degree in electrical engineering and her master’s degree in business administration from the University of Florida.