Archive for October 2011

Mobile. Mobile. Mobile.

Posted by Tom Lounibos on Oct 18, 2011. Leave a Comment

It will come as no surprise to any of you that after years of projected success, the worldwide mobile app market is actually beginning to reach expectations.

With the emergence of a whole new suite of smart devices and development and deployment platforms, everyone is racing to reach consumers through mobile apps. This is welcome news for those of us in the mobile space and especially for players patiently waiting for the market to take off.

Keynote Systems has been pushing mobile monitoring since 1999 with only marginal success. DeviceAnywhere, meanwhile, built a very expensive physical network of global devices and providers for the purposes of monitoring mobile apps in 2003 and has been treading water waiting for the mobile app market to emerge. Well, the market is now cooking; good for both of these companies for holding on during the tough times. The question is, at what price did they pay by doing so?

Last week they announced a merger. Why now, just as the market is beginning to evolve? For Keynote, the merger makes the most sense because its approach of “emulating” mobile devices has come under fire from more than one source. By acquiring DA it will improve its approach significantly. However, is this enough?

Some think that DA’s own technology has some very real limitations.  Mobile monitoring and testing just has not been able to keep up with either the technical advancements in dynamic content or with the agility in which these new technologies are being applied today … and subsequently the performance of mobile apps has been terrible.

I applaud the tenacity of these two great companies for persevering over all these years as the mobile markets matured.  That said, I predict that the coming years will see a real renaissance in the mobile app test market that will ultimately change this game once again, thus thrusting a whole new generation of mobile test and monitor vendors into the forefront.  Should be exciting times for everyone building mobile apps today.  For Mobile Dev/Ops/Test market … “These Times, They are a-Changin!”

 

I’m Back!

Posted by Tom Lounibos on Oct 12, 2011. Leave a Comment

What does a Cloud CEO do for a summer vacation?

It’s been a while. According to my new editor-in-chief, Leslie, I have not blogged since mid-summer. I do tweet regularly, though, at @lounibos. The fact that she’s not too happy about means it’s time for us to catch up.

What’s been keeping me away from blogging? Was it the lure of the beach? Nope! Was it my love of baseball? Nope! It was what it always seems to be for fast-growing Cloud startups: fund raising! Yep, I spent the summer rummaging around the Venture “Forest” that we call The Silicon Valley.

However, this time was very different as I had not planned to forage for cash until the spring, when fresh capital begins to bloom. This time we got started under the heat of the summer sun because, unexpectedly, a private VC decided to go rogue and preemptively offer us capital in which to grow. That’s how it all began.

I’ll tell you the rest of the story and more about our growth in time. And so here we are, 90 days later, and I’m emerging from the forest realizing that the summer has passed and that it’s time to update my blog.

That’s the short version of how I spent my summer vacation, along with the general business of talking to our customers, prospects and employees about their challenges and how we can solve them.

Leslie has made me promise not to disappear again!

 

Stop Cheating on your Tests!

Posted by Dave Murphy on Oct 06, 2011. 2 Comments

I suppose we could have used a less inflammatory title for our recent webinar. It makes it sound like testers have been purposely doing something wrong. Perhaps we could have titled the webinar “Now you can execute more accurate and informative tests!” But the folks in marketing were right, and the intriguing title attracted our largest group of attendees ever. For those of you who didn’t attend, or if you did and would like to review the messages, you can watch the webinar here. This was the first in SOASTA’s latest webinar series, “Cloud Testing – Rewriting the Rules of Performance Testing”. Future webinars include “Run More Tests and Find More Issues” on October 27th and “Test On Your Schedule across the Lifecycle” on November 15th.

In this webinar, Scott Barber, President and CTO of PerfTestPlus, joins SOASTA’s VP of Performance Engineering, Rob Holcomb to discuss what performance engineers have done in the past to measure performance and find and fix issues; and why some of those techniques no longer reflect best practices. The focus is on web and mobile testing and why the higher scale, more distributed and often complex nature of that traffic is not well served by traditional testing tools or techniques.

After an introduction by SOASTA’s Brad Johnson, Scott, in his inimitable style, speaks to great effect about the four most common ‘cheats’ that performance testers have leveraged to overcome the constraints of inflexible test hardware, poor tool scalability, expensive pricing models and the lack of real-time information while testing. Scott begins his presentation by talking about the practice of modifying think times, typically to overcome licensing and/or hardware limitations imposed by the high cost of traditional load testing. His primary assertion: the only way to simulate production…is to simulate production. Interestingly, during the webinar a question came up suggesting that we’re testing computers, not humans, so why is accurately simulating user activity so important. In response, it was noted that it has become clear that variance in use absolutely has an impact on what happens to the infrastructure.

The second point discussed is the common practice of extrapolating results from a staging environment to predict what will happen in production. Architectures can be complicated, and the impact of those differences along with the additional complexities of ‘the real world’ make extrapolation problematic, at best. The best way to validate that your production environment will handle expected load is to test in production as part of your overall test strategy. (For more on testing in production check out this SOASTA webinar). Modeling user flows incorrectly is the third point addressed by Scott, reinforcing the notion that we’re not functionally testing the application, but need to make sure we’re putting a realistic load on the back end.

Finally, Scott presents a very interesting problem to illustrate the challenges associated with measuring performance, and how it can be as much an art as a science. Rob follows Scott’s presentation and, using SOASTA’s CloudTest, illustrates how we can use modern tools to, well, stop cheating. We hope you enjoy(ed) the webinar.

Email Us!
Subscribe to our Feed!
Find us on Facebook
Follow our Tweets
See our pics