Going Private is the New Black

Posted by on

Written by Tom Lounibos

February’s announcement of Dell’s move to go private got me thinking. The deal, put together by Michael Dell and Silver Lake Partners, is potentially worth $24.5 billion or more depending what type of impact disgruntled investor and take-over czar Carl Icahn has on its negotiations.

In the meantime, BMC has announced and closed a deal with two private equity firms worth $46.25/share in cash or $6.9 billion.

Who will be the next tech company to close the curtain on public investors … Compuware, CA or even HP? And why the move to go private?

GigaOM tells us that cloud is the disruptive force at work:

As more companies evaluate the economics of putting workloads on massive webscale infrastructure — outside their walls — they will buy far fewer servers and routers themselves. And as more corporate applications are delivered via software-as-a-service models there are fewer huge upfront software licensing deals. Instead payments are spread out across a year or three. There is also pressure on the massive enterprise service and maintenance fees favored by companies like Oracle.

“There’s a seismic shift afoot with enterprise software vendors as they move from traditional pricing and distribution models to OpEx, SaaS and cloud models. This means a financial disruption for many of them, not just BMC Software,” said Dana Gardner, principal analyst with Inter-Arbor Solutions and GigaOM PRO analyst.

Seismic shifts in tech seem to be a continuous exercise. This particular shift, driven by an ever-growing mobile / social world, is of such an EPIC proportion that even industry veterans have never seen one like it before. Disruption is being experienced at every layer (User, Device, Network, Infrastructure) of the technology stack. This means that these are not good times for the incumbents (public vendors) whose technology and business models are showing their age. They will need to evolve or die, and evolution is best done in private. Innovation can be a messy business for even the most confident investors. It has many highs and lows, which makes it difficult to be watched by a glaring public eye. And so going private has become the new black in tech.

Nor is it just Dell and BMC. Other companies such as Compuware, HP, IBM, Oracle and Microsoft have angry shareholders who are unwilling to face the fact (or do not yet grasp) that the evolving product set and buying pattern means that profitability models are shifting, too.

Nor is it just tech. Berkshire Hathaway and 3G Capital are spending $23 billion to acquire HJ Heinz Co. in what is thought to be the largest acquisition in food industry history. Best Buy and Barnes and Noble have been mentioned in the press as considering going private.

What does this shift to private mean for all the newly minted private tech companies that dream of someday going public? Time will tell, but one thing I’m sure of in tech is that those who innovate faster typically win!

Mix Master CT Kicks Down with Dynamic Ramp

Posted by on

studio

You may have never been a DJ  or “Spin Doctor”, but if you ever worked a graphic equalizer or used digital music mixing software, the power of the sliding controls to fade sounds, control volumes or change balance is familiar to you.

Imagine controlling web and mobile user traffic driving millions of users from multiple geographic locations or increasing the frequency or scale of realistic user scenarios with exactly the same controls!

The SOASTA CloudTest product team did!  It’s called Dynamic Ramp, and not only is it incredibly powerful, but it’s really fun to use!  More importantly, it represents the continued shift in real-scale performance testing where teams have moved load testing outside the test lab into production environments.

One of the most rewarding aspects of seeing the CloudTest platform in the hands of users over the years is to see innovative applications of the technology.  From early on, the cloud offered more capacity for load generation than most customers could handle in test labs.  This meant that more tests were run on production systems during maintenance windows or in slow hours.  With instantaneous feedback to system performance, customers began exploring with larger tests, and targeting new goals like peak capacity, effects of rapid spikes and sustained traffic at high levels.

During live tests, customers often asked if they could increase or decrease traffic or add more or less from a particular location as tests ran. It was always easy to pause or stop a test, modify the load levels and restart it, but we soon reached critical mass for enhancement requests for live ramp control, and the SOASTA team spec’d out the feature.

dynamic-ramp

CloudTest’s Dynamic Ramp feature takes SOASTA’s already popular digital music mixing metaphor to the next level.  Users can now modify running tests on the fly.  Slider controls enable the scaling up or down on a master level, or scenarios can be altered individually with single fader controls. This makes anything possible in a single test session!  If all is well, let’s ramp it up!  How about add a spike of traffic from London?  What if we’re testing in production and response times creep up to risky levels?  Dial it down and keep on going.

Performance Engineering has never been more important, or cool, and CloudTest continues to benefit from the evolving use-cases of customers pushing the envelope in pursuit of end user satisfaction.  Take a look at the new video of this terrific feature and get ready to rock your own master track with CloudTest!

SEE IT NOW!

Guest Post: Mobile and Cloud Converge to Change the App Landscape. . .

Posted by on

Mobile and Cloud Converge to Change the App Landscape . . .

. . . What Does that Mean for your Business?

This guest post is authored by Jeff Vance, the founder and Editor-in-Chief of Startup.50, a site devoted to identifying hot new startups.  Jeff also regularly contributes feature stories to CIO, Forbes.com, Network World and many other publications.

Smartphones are taking over, rapidly pushing desktops off of their perch as the go-to computing device. According to information gathered from telcos and network operators and compiled by consulting firm Chetan Sharma, smartphone penetration in the U.S. passed the 50 percent barrier (versus feature phones) for the first time last year.

A study by Litmus found that 38 percent of emails are now opened on mobile devices, and ComScore’s new Mobile Metrix 2.0 report found that most users prefer to access social networks from smartphones, and when they access these networks on their phones, they prefer to do so via apps rather than a browser.

At the same time, cloud computing is quickly becoming mainstream. It’s catching on quickly with consumers (Dropbox, Google Docs, Spotify) and businesses alike (Box, Evernote, Salesforce.com, Marketo).

As these trends converge – and they are indeed converging, since the cloud helps mobilize apps, protecting constrained devices from heavy on-board processing and large amounts of device-side storage – many pundits are worrying about how this will affect businesses productivity.

Mobile apps usage is driven by a quality user experience

To ensure success for mobile apps, users need fast, affordable, comprehensive application testing. According to a study by Nuance Enterprise, while Android and iOS users now download about 10 apps each month, 95 percent of apps are abandoned within a month of download. The two main reasons apps were abandoned were slow response times (to launch, load or connect) and poor usability.

Developers are under extreme pressure to mobilize existing apps, create new mobile apps, and port various mobile apps to new platforms. In the process, many, many apps get into the hands of consumers well before they should.

Personally, I uninstall most mobile apps after trying them for the first time and finding that they freeze, crash my phone or just don’t perform well. If I uninstall apps more than once from a specific developer, there’s a good chance I won’t ever download their apps again.

This isn’t always the fault of developers. According to the initial findings from an ongoing Aberdeen survey, The Challenge of Application Performance in a Mobile Application World, providing mobile access to existing applications is the number-one app-related concern for nearly 70 percent of organizations. Yet, only thirty percent have implemented any kind of mobile application testing and performance assurance infrastructure.

In other words, the emphasis is on getting apps out the door. You can worry about performance later. The sad truth is that this is consistent with the history of software development, where products are shipped with scores of known bugs and developers then release endless updates and patches.

Lack of testing could alienate users – forever

Plenty of app developers do test but do it in a rushed way. Many developers believe that they only have the time to run limited lab tests. The pressure to get something, anything, out there is so great in the mobile world that what used to be alpha or beta releases are now published as if they were the final release. Many developers believe it is okay to put apps into app stores – especially if the app is free – in order to attract early adopters to serve as testers – often without the knowledge that they are serving as testers.

I downloaded a billiards app, for instance, to help me track my shots and determine which ones I missed most often. Then, I’d know what to work on in practice sessions. In its current iteration, the app stinks. I’ve given the developer feedback and I’m willing to give the company a chance to work out the kinks – but only for two reasons.

One, there’s really not another app like it in the Android Marketplace. Two, the stakes are low.

If there were a different option on the market, I would have asked for my money back and moved on. If this were a business-critical app, I would have uninstalled the app, given it a bad review and searched for an alternative.

Not everyone gives even low-stakes, free apps the benefit of the doubt because user expectations keep rising. High-quality free apps from Google, Facebook, Dropbox, Hootsuite, etc. mean that users expect more.

User experience is driving the success or failure of apps and usability is now a major competitive advantage.

Mobile testing must involve real-world scenarios

Fortunately, the industry is waking up to the need for more robust mobile testing. A couple of months ago, I evaluated more than 150 mobile startups for Network World’s “10 Hot Startups to Watch” series. The single biggest sub-segment was testing and monitoring, with close to 20 startups.

Most of these companies understood the importance of testing, and most were leveraging the cloud to make testing cheaper, faster and available on-demand. But the overall philosophy of testing still hearkens back to the early days of software development, when apps were tested in a limited lab setting and then shoved out the door.

That just won’t cut it in the mobile age. The only way to effectively test an app is to test it under real-world pressures. With mobile apps, this is a huge challenge since ensuring mobile app performance means you’ll need to account for different platforms and devices. If you don’t test in a real-world production environment, then you are turning your early adopters into de facto guinea pigs.

It’s fairly easy, though, to make your early adopters partners, rather than guinea pigs. A limited, invitation-only beta release will attract tech-savvy users who will want to push apps to their limits. This is much better than a sterile lab test.

However, early testing is just the start. App testing is an ongoing process, not a one-off project.

When an app doesn’t work, users don’t have the patience to wait for you to improve it anymore. They’ll just go elsewhere because this is no longer the age when a handful of software companies have near monopolies. Today, users have choices.

Conversely, if you deliver a tested, high-performing app to the market, users appreciate each and every improvement. If you streamline the interface or add a new feature, you improve customer stickiness. You only get this goodwill, though, if your app works as advertised from the very start.

Ongoing monitoring and testing isn’t just about keeping the app stable and functioning, though. In the mobile age, user behaviors change constantly. Monitoring apps means you’re monitoring user behaviors.

You’ll know if users typically leave your app to go to a mapping app, which would mean you should offer one-click integration with that app. Or, perhaps, users constantly share data in your app on social networks, which could show you the opportunity for a new sharing feature or even an entirely new app.

There are so many apps and users have so many choices now that app testing is mission-critical and must encompass the entire lifecycle of the app. In other words, the app lifecycle should be modified to be more of a continuous loop:

-Development phase that includes deeper levels of test automation
-Testing phase that includes a range of devices and realistic user scenarios  (alpha and beta releases)
-Deployment
-Ongoing monitoring and testing of production systems
-Collection and analysis of real user data
-Ongoing app improvements and enhancements based on user data
-New product development based on user data

Developing new apps is no longer about educated guesses about what users want. Now, you have real user data to guide your decisions.

Guest Post: An Experience Report Building a Mobile Automated Test Lab With SOASTA TouchTest

Posted by on

Written by Wil Pannel. Wil Pannel is an enterprise Software engineer evolving as an agile practitioner. His firm, Media Agility, provides consulting to global clients seeking the latest approaches to mobile software development and deployment.

Prelude

I came to develop a PhoneGap application on IOS and Android tablets. Agile/lean engineering practice led to test-first development, but when I began to build out continuous integration, I found the offerings for mobile automated functional testing and deployment lacked maturity.

I discovered SOASTA when I attended a webinar on mobile functional testing that culminated with a photo of a mobile lab, to which a continuous integration process was deploying and running an automated suite of functional tests to a variety of IOS and Android targets:

I required no further inspiration.  Such an outcome would provide my organization with a high degree of code coverage for our mobile offerings and protection against defect regression — to build the product right — and establish an infrastructure to deploy feature changes continuously — to build the right product.  The doors to lean mobile platform development are now wide open for us.

SOASTA and CloudBees

The webinar was hosted by SOASTA in partnership with CloudBees, a cloud-based provider of Platform-as-a-Service for developing web-based and mobile applications. We followed the practices recommended by SOASTA and CloudBees closely, and built out our Jenkins deployment pipeline comprised of the following jobs:

1. build the javascript assets for deployment to the IOS PhoneGap project;
2. build and deploy the IOS app capable of running SOASTA functional tests to devices tethered to the mobile lab;
3. kick off SOASTA functional tests;
4. build and deploy the production-ready IOS app to our enterprise Appaloosa store and notify QA that a new release is ready for a spot check.

We also built out an analogous pipeline to deploy and test our Android PhoneGap project designed to run concurrently with the IOS pipeline.

Features of Our Mobile Lab

(1) We run over 350 automated unit- and integration-tests and achieve greater than 80% code coverage:

(2) Using SOASTA command-line tooling and the Jenkins XCode plugin, we build a mobile-functional-test-ready IOS target and deploy it to the devices tethered to the mobile lab:

(3) Again, we use SOASTA command-line tooling to run a pre-recorded mobile functional test on a SOASTA-hosted instance of their TouchTest environment.

(4) Finally, provided each job succeeds, we build and deploy a production-ready release to our Appaloosa private enterprise store:

(5) There is a step that I purged, which uploaded to a QA site for approval.  This meant that the build would not pass until QA exercised their exploratory test phase. I did not feel right about the way in which this step would prevent continuous deployment.  But I felt that it was right to purge this step after discussing the topic with Joshua Kerievsky.  He asked me a simple question:  can the automated test do exactly what QA would do?  Because of the accurate recording I experienced with SOASTA TouchTest, my answer was:  yes.

Quick Starts and Support

There exists a ton of practice on both SOASTA and CloudBees for getting a mobile lab up and running.  SOASTA’s quick starts and knowledge base are comprehensive, and I found the forums and personalized sales support via email extra-ordinarily responsive.

The same is true of CloudBees.  Their partner demo site and blog provided concrete configuration and practice for many of the scenarios we required.  And Mark Prichard of CloudBees himself generously devoted a considerable amount of his time to walk thru the fine points of my Jenkins configuration.Summary

The other offerings I discovered that support mobile functional testing had obvious shortcomings in one-way-or-another.  SOASTA is the only one, to my knowledge, that can support development at enterprise scale. My favorite feature is to be able to deploy from the command line to IOS hardware targets in a continuous deployment scenario without manual intervention.

At the time of posting this, I learned that SOASTA has shipped my most-anticipated forthcoming feature:  the one that wakes up the sleeping device tethered to the mobile lab when the automated build deploys.

Our experience with SOASTA tooling for automated mobile functional testing was quite satisfying.

No, thanks, I really did want prototype

Posted by on

In CloudTest, we’ve built an error capture and reporting process similar to what you see in Apple’s crash reporting.  If we detect an error, either on the server or in the client’s web browser, we’ll show an error dialog which allows the user to supply additional information and submit the error to SOASTA so our engineers can fix the problem.  It’s been immensely helpful to find and diagnose bugs.

As an engineer, this can be a blessing and a curse.  It means you’ll get lots of bug reports very quickly if you’ve got a problem.  In Java land, on the server, this usually is great.  We can capture detailed stack traces and attach the application server’s log files to the bug.  This drastically reduces the time to research and reproduce bugs.

In JavaScript land, on the client’s web browser, results vary.  Currently the best method for getting client-side errors is to use window.onerror.  It’s not perfect.  It doesn’t catch every error, and many times catches harmless errors you’re not interested in.  Additionally, the information that’s captured isn’t always as rich as server-side (Java) errors, and many times doesn’t contain a stack trace at all, rather just a single line of text.  This generally leads to bug reports that cannot be reproduced, thus making a fix very difficult or impossible.

Recently, we’ve noticed an increase in a certain “class” of errors from JavaScript.  The code base for CloudTest was started in 2006.  At that time, jQuery didn’t exist yet, the initial release came a few months after.  However, Prototype JS did, and was judged to be the best option for our needs.  We folded it in and have never looked back.  If we were to make that decision again, we’d probably go with jQuery just because of the widespread adoption and community support.  Still, to this day, Prototype JS is a great framework, and I’m always happy to use it.  I’ve never found anything in the base jQuery library that Prototype doesn’t handle in an albeit different manner, but equally adept.

Now – getting back to that comment about widespread adoption.  It would seem that many browser extension developers believe that jQuery really should be the one-and-only JS framework to exist on today’s modern web.  So much so, that they inject jQuery into the current DOM, without using the noConflict() method!  For many sites, this isn’t a problem.  However, if your web browser is on a website that already defines any function in the global namespace that jQuery also defines, jQuery will overwrite it.  This isn’t a specific problem with jQuery, rather a problem with how browser extensions are written in some cases.  For our particular problem, if a website (such as CloudTest) uses a JavaScript library (such as Prototype JS), which defines a function in the global namespace (such as the $ function), and a browser extension injects some JavaScript (such as jQuery) which defines the same function into the global namespace, your code will be overwritten.

Obviously this is a very powerful tool, and is the crux of extensions like Greasemonkey.  However, when wielded incorrectly, it can cause disastrous effects.  Imagine a website that has code written specifically for Prototype JS, with lots of calls to the $ function.  In Prototype JS this is a shortcut for document.getElementById().  In jQuery, the $ function returns the global jQuery object.  If, for instance, you want to set an attribute on a DIV with id “hello”, you might write:
$(“hello”).setAttribute(“message”, “goodbye”);

When this code executes after Prototype has been overwritten with jQuery, instead you’ll receive an error similar to:
Object [object Object] has no method ‘setAttribute’

This proved to be one of those non-reproducible head-scratchers for quite some time, until we could view this on a customer’s browser first-hand.  5 minutes of debugging showed a BitTorrent browser extension was spamming the entire global DOM with jQuery, after the DOM had loaded.

So – what to do?  We aren’t going to go back and rewrite hundreds of thousands of lines of code to retrofit for jQuery, just so a customer can use a BitTorrent extension with CloudTest.

Instead, it’s a simple matter to detect this and give a friendly message suggesting the user disable browser extensions.  In our error capture process (from window.onerror), there’s now a simple test:

if (typeof jQuery != ”undefined” && jQuery == $)
{
// show a friendly message and return.
}

Hopefully this will lend to less time scratching heads and more time writing cool new features.

Guest Post: Why Mobile App Testing Requires a New Approach and How We’re Doing It

Posted by on

This guest post is authored by Simon Berman, Sr. Director, Product Marketing at Appcelerator

Today, as we were delivering our live webcast to the business press, industry analysts, technology bloggers, and enterprise customers, I was watching the mobile testing demo with great interest. After spending over half my career in automated testing (12 years at testing pioneer Mercury Interactive), I couldn’t help but notice the similarities with our old 90’s era WinRunner with its nifty record/replay technology, verification points and context-sensitive test scripts.

But then it also struck me that there are some fundamental differences between the testing of the business processes of applications like SAP and Oracle for enterprise (desktop) users, and today’s mobile apps. Sure, the underlying technology today is different, as is the delivery mode (now via the cloud) and the user experience, but there are more substantive ones too:

1. Testing vs. Quality

Testing in those days was never considered a glamorous discipline. QA used to be considered low on the corporate totem pole; the engineer wannabees, the naysayers, the ones who invariably stood in the way of releasing new products and services to market. The business would commonly shove in some more features to the release expecting that users would thank them for giving them so much functionality! “If it doesn’t work, we’ll patch it in the next release” was the mantra.

The CEO certainly didn’t know much about Testing either – if anything, it was just an impediment standing in the way of delivery. So are we really surprised that over 50% of all IT projects failed to be delivered on time, on budget and on requirements?

Today, I guarantee you that every CEO whose company has one or more mobile apps (whether for customers or employees or partners) is keenly aware that the quality of those apps is of paramount importance, and can make the difference between successful adoption and failure. They may still not know much or care about testing, but they certainly do care about Quality. The quality of the user experience, the design, and the function is now the overarching theme that takes precedence over everything else. Better to keep the app brain-dead simple with high utility, than a monster app with low usability. Quality is now an integral part of the delivery lifecycle, and QA (Quality Assurance) has evolved to QE (Quality Engineering), with a direct impact on the success of the business.

2. Release Cycles: Months vs. Days

When enterprises rolled out new versions and modules of their ERP/CRM systems, release cycles were literally measured in many months, even years. It wasn’t uncommon for a U.S. rollout to take 18 months, followed by another 18 months for subsequent regions. Changing timelines from the business to roll out new versions, coupled with engineering feature-creep meant that testers were constantly struggling to figure out what to test and when to test it.

In a previous blog (“Nothing is certain except death, taxes and a short mobile app lifespan”) we discuss how mobile enterprise apps need to be focused on a small feature set targeted around a specific set of needs to be successful and that the best approach is to write apps that users want and need right now. Most likely these apps won’t even live more than a year or two in total, and that’s after going through multiple iterative releases in between. So from a testing perspective, the feature set is much more defined.

Today, with agile development practices release cycles are down to days or even hours. With cloud development and DevOps teams, the line between development and testing has become blurred. Everyone is developing and testing at the same time, and the traditional model of testing purely as a discrete event prior to release is no longer valid.

3. Desktop vs. Mobile Devices

How simple life was when we had a corporate standard for the enterprise desktop; one brand of manufacturer (Dell, HP), one operating system (MS Windows), and one certified browser (Internet Explorer) – all governed and controlled by IT. Worst case was that there were a few supported browser versions, but by and large the number of end user platforms on which the apps would run was very small. This made the testing far more manageable since the number of permutations was so small.

With mobile today, there are literally hundreds of permutations to test for based on device manufacturer, platform, O.S., versions and display size. Factoring in the Android fragmentation (iOS fragmentation is also starting) and pretty quickly the list of testable permutations starts to get messy. On top of that, enterprises require a legitimate solution for testing that doesn’t involve jail-breaking or backdoor tomfoolery.

Requirements for Mobile Application Testing

The automated testing of today’s mobile apps therefore needs to be suited for this new world. The key requirements include:

-Automated capture/replay (without jail-breaking)
-Support for multiple platforms and device types
-Integration with the development environment for rapid and continuous testing

Today Appcelerator announced a partnership with SOASTA the leader in cloud and mobile testing as well as a joint integration between Titanium and SOASTA’s TouchTest. We’re also now selling this solution under the name Appcelerator Functional Test as a new part of our platform. This, we believe, will provide our enterprise customers with a strategic advantage in their pursuit of business-transforming mobile apps.

Here’s what George Mehok, CIO of Safeguard Properties, the largest mortgage field services company in the U.S, had to say about this new solution and why they’re adopting it now:

“We’ve adopted mobility as a key enabler to providing productivity enhancing technologies to our nationwide network of contractors and inspectors, in addition to delivering superior quality to our clients. As an innovator in mobility, we’re actively implementing the integrated solution between Appcelerator Titanium and SOASTA TouchTest to further optimize the quality of our apps, reduce development cycle time, and accelerate adoption.”

If you’re interested in learning more about this, watch the recent webinar.

describe the image


Categories: Mobile

Could developers be the future of software testing?

Posted by on

According to different reports, the global software testing market is around $30B and is looking at growing at about $50B in 2020. Not too shabby! It is estimated that around 30% to 40% of this market is taken by offshore testing services (these days located in India, Eastern Europe and South America for the most part). I have nothing against outsourcing testing services as long as they bring the best value to the development organization. The problem with these services (from my own experience) is the fact that they still operate the same way than 20 years ago. Basically performing manual testing at the end of the whole development phase and have still a lot of difficulties to embed within an agile environment. I’m thinking that a lot of testing services company (especially the largest one) are doing everything they can to keep software testing in a status quo, in a dormant state for their own benefit. I don’t see a lot of these companies trying to invest in automation offering. How long will they keep fooling customers?

fred 1

 

A new study by ABI Research is bringing me some hope. According to them:

The revenues from mobile application testing tools will exceed $200 million in 2012, finds a new study from ABI Research. About three-fourths of the current revenue base comes from tools that enhance manual testing, yet over the coming years the market will be driven predominantly by test automation.The growth from the automation segment will push the revenues close to $800 million by the end of 2017.
[...]
What makes testing mobile software more demanding than testing computer software is the sheer complexity of the environment, which stems from the diversity of relevant devices, operating systems, and networks. Furthermore, for consumer-facing apps the low barriers to entry and the intense competition over users’ attention mean that the first impressions are critical for any app’s success – and there can’t be good first impressions without adequate, rigorous testing processes. For employee-facing enterprise apps the cut-throat competition isn’t an issue, but in their case a poorly conducted quality assurance can result in user inertia and productivity losses.

What’s driving more a more companies toward test automation is also the ever shrinking release cycle they’re facing for their mobile apps. Whether it’s a mobile web app which can see multiple versions per day, or a native/hybrid app which can be released any other week, the need for continuous feedback is critical.

I was at the Appcelerator Codestrong event where I had a chance to discuss with dozens of developers. Their main concern with traditional testing is the lag between a new build and the feedback they will get from testers. They want to be able to fix their bug within hours of their code check-in, while their mind is still in the code. This is why more and more developers are looking at investing some of their time to automate their own tests, making testing part of the continuous integration process and one of their responsibilities. They are writing unit tests for their mobile apps but from what I’m hearing at such conferences is the fact that they’re ready to expand their testing , bridging the gap between unit testing and what traditional testers can offer. I truly think this is one of the reason the automation market is going to grow so fast. I still don’t see traditional offshore testing services investing into these kind of services, still trying to protect their very lucrative manual testing market. But I do expect to see a lot more tools and services built around developers requirements rather than traditional testers requirements.

There have been a lot of articles stating that “Test is Dead” over the past year (Ever since the keynote at GTAC 2011). I don’t think testing will ever disappear (nobody does) but it looks like the explosion of mobile could (finally) change the game on how we perform testing and will bring a lot of innovation to this “bastard child”. What’s very interesting is the fact that developers will probably be the driving force for this innovation. My money is on them!

What do you think? Do you think there is a shift in the software testing market toward developers?

The only way to conduct real-world tests … is in the real world.

Posted by on

This guest post is authored by Seth Eliot, Sr. Knowledge Engineer in Test at Microsoft and a well-known speaker and blogger. Seth shared additional content during an October 31 webinar with SOASTA on Testing in Production Advances with Big Data and the Cloud.

In 1985 Microsoft ran a classified ad in the newspaper recruiting testers:

As a Software Tester you will design execute and document tests of application software. You will generate test scripts and automatic test packages. This is a challenging and highly visible position within a fast growing division of Microsoft.

Read more on “The only way to conduct real-world tests … is in the real world.” »

The Epicenter of Scale

Posted by on

I’ve just returned from a milk run through Hong Kong, Beijing and Shanghai.

China, and Beijing, are redefining all western views of scale. The business diversity is immense. It’s hard not to be impressed by the curiosity and enthusiasm of the citizens of the world in this very modern city, in this very modern country. The sheer size of the physical landscape and architecture is overwhelming. I liken this to a 100-mile drive through Manhattan.

 

 

Some facts for you:

  • China is the fastest growing market for American goods. Between 2000 and 2011, US exports to China rose 542 percent, while its exports to the rest of the world rose by only 81 percent during that period. (Forbes.com)
  • In 2008, the Beijing Capital International Airport opened Terminal 3, an addition that made the airport the world’s busiest. The Beijing Daxing International Airport, to open in 2017, will have room for 130 million passengers each year. (Australian Business Traveler) In comparison, New York’s three airports have a combined capacity of 110 million.
  • After 7.5% growth in 2012, estimates have China’s economy growing between 8% and 8.5% in 2013.
  • There are 84 cities in China with a population of more than 1M people; in the US there are nine. There are four cities in China larger than any US cities.
  • On Singles Day, Alibaba’s TaoBao.com, the Chinese equivalent of Amazon, took in over US $3B in revenue, beating the total of ALL US retailers for CyberMonday 2011 by $1.75B.

And finally, in 2006, after 30 years of trying, the Rolling Stones, arguably the world’s most famous rock band, made it to China, after a long negotiation which included some sensorship. At one point in the concert they shared the stage with Cui Jian, a Chinese musician who said, “This is the 20th-year anniversary of Chinese rock ‘n’ roll,” referring to the rock ‘n’ roll era’s bypass of China, which was in the midst of its Cultural Revolution during the music’s heyday.

As I was reminded again last week, this land in the midst of great change and rapid market transformation offers incredible opportunities for those that can navigate the complex mosaic of culture and business.

Is your Enterprise QA organization ready for Mobile?

Posted by on

Unless you’ve been living in a cave for the last 18 months, you should know that Mobile is the way forward to conduct business, deliver media and entertain. There’s still a lot of debate about whether native apps are the right model (according to Marc Andreessen, The application model of the future is the web application model), if HTML5 is finally going to take off or if a hybrid approach represents the best of both worlds. In any case, no one can argue that mobile is just a fad.

At SOASTA, we talk daily with companies trying to figure out their mobile strategy and how they can deliver the best quality apps to their customers. These discussions almost always result in the same conclusion: a majority of companies have been taken off guard by the mobile world and are struggling to transition their QA department to support this new model. It feels like the late ’90s all over again when companies were trying to figure out how to best transition to the Internet era. The mobile era brings somewhat similar challenges, but the requirement to transition is exacerbated by the speed of mobile adoption, proliferation of smartphones and mobile devices, roll-out of 4G and an explosion of social media.

According to SOGETI’s world quality report, only 31% of companies across the globe test mobile applications. When you consider that between the two major players, iOS and Android, there are more than one million apps available, that’s a lot of apps being released without any type of testing! What are some of the main reasons for this situation?

The right product

65% of companies are still trying to figure out what is the right product to test their apps. This is due to the fact that, as they figure out their mobile strategy, they realize that the list of requirements for their testing product gets larger. At first they only had to support iOS. Then they realized that Android was also key. Then they wanted a mobile web version or maybe hybrid. They had to think about performance of their front-end, then the mobile app back-end. A lot of testing products specialize in a few requirements and organizations sometimes ends up with 5-6 different products for all their testing needs. This might be a short-term strategy but is not sustainable in the long run and can’t really scale, especially in an enterprise environment where standardization and integration is a must. Many companies that we talk to are very interested in our “platform” approach, allowing them to cover all their requirements with one integrated product.

A new Infrastructure to manage and leverage

52% don’t have access to the required devices. Fragmentation is a critical challenge faced by organizations and one they can’t ignore. But a lot of them are still trying to understand how to manage it. By discussing this with customers, we realize that they don’t invest into devices not so much because they don’t know which one to pick, but because they struggling to integrate them into their development and test life cycle. At SOASTA we’ve been tackling this challenge from Day 1 of the development of our mobile solution, TouchTest. We’ve been building our mobile lab to support the testing of our own product. Today, for each build, thousands of tests are run automatically in our iOS and Android mobile lab. New versions of the mobile apps we use for our testing are deployed automatically on these devices, tests are launched by our continuous integration framework (Jenkins) and test results for each build are available in dashboards along with all of our other tests (unit tests and performance tests). We’re happy today to be able to package these labs and enable customers quickly and affordably.

Skills and processes

According to the World Quality Report, “a majority of organizations characterize their internal teams as ‘average’ at best in their knowledge of core testing processes and methodologies, and not necessarily up to speed with the latest testing tools and technologies” and “less than 5% of firms are fully confident that their testers (internal or external) are ‘best in class’”. Testers have to face the same challenges they had to tackle during the web era: supporting legacy applications running on the desktop (or mainframe) while investing some of their time to transition to the web. Today they have to go through similar transition at a much accelerated pace.

The report also mentions the fact that a third of organizations they’ve talked to lack the testing methodologies and processes necessary to effectively certify mobile applications. Any new technology brings new methodologies and processes, and mobile is not any different. We found this out very early during the development of TouchTest and we’ve been building our own methodology for the last few years now. A couple of weeks ago we decided to share our expertise and methodology by creating a mobile division to help customers set up their own Testing Center of Excellence (TCOE). According to the report, Enterprise companies are more and more interested in industrializing their testing activity and are looking at TCOEs to act as a virtual command center, using a standardized approach and a flexible pool of available resources. According to the report, 60% of companies plan or are developing a TCOE (up from 45% last year). By providing mobile test automation and performance experts, continuous integration test methodologies and management services for internal and external mobile test labs, we hope to be part of their success.

Conclusion

Our own observations and the results of the World Quality Report clearly demonstrate the challenge organizations are facing today. QA needs once again to reinvent itself (and follow the steps of development, which is almost always faster to innovate and to keep up with technology) and find its place into the mobile application lifecycle. In a more and more agile world with shrinking delivery cycles, QA needs to find ways to bring continuous value very early and throughout the whole lifecycle. Organizations should make sure testers have an opportunity to be properly trained to this new mobile landscape.

Additionally, the new mobile infrastructure needs to not only be properly evaluated but also completely integrated within the development and test cycle to bring its full potential. The mobile device is the new desktop, the new server, the new browser. A small difference is that the scale of complexity is twenty-fold: consider the type of OS, version of OS, type of devices, size of devices, the brand, different mobile web browser, different technology for the apps, etc.

Finally, QA organizations need to figure out very quickly if they’d rather invest time and money in dozens of different products or if they want to settle on an integrated testing platform dedicated and built from the ground up for mobile. Testers have a tendency to invest a lot of their time evaluating new products which support just one requirement. We recommend that they take a look at mobile more broadly and understand how their company is going to leverage it in the future. Will there need for testing iOS? Android? Both? Mobile web or hybrid? How about performance or real-time monitoring? Is there a requirement to understand the impact of mobile back-end stress on the front-end of the mobile app? Or a need to run thousands of tests for each build to guarantee a weekly update of the app? These questions are important to answer before investing in any testing products as a bad decision is not only costly but would also reinforce the perception that QA is the bottleneck.

Did Affordability Save the Holidays?

Posted by on

We’ve just emerged from the largest US holiday opener in online history – Black Friday and Cyber Monday 2012! Generally, the web performed favorably. Congrats to the well-prepared!

  • Cyber Monday sales were up 30% from last year, just shy of $2B – $1.98 – from Internet Retailer.
  • Black Friday online sales exceeded $1B for the first time ever according to Reuters.
  • Mobile sales were up 100% amounting to $640M over the two days.

Rounding to whole numbers – that’s $3B in online sales in 2 days!

(Note – Chinese online retailers did $3B in 1 day this month…but that’s another blog.)

The statistic that makes for many proud SOASTANs is that this list from Black Friday’s top performing sites is a who’s who of our customers who prepared with CloudTest! Hooray to them all for legendary performance! Not surprisingly, there were reports that mobile web user experience took a big hit. Might that $640M have been higher with better mobile performance?

Now that “web scale” includes mobile access, testing approaches must embrace the expanded requirement to cover mobile performance. At least for our customers, affordability has had major impact in allowing them to test frequently at realistic scale and include mobile user profiles that were out of reach in the past. It is heartening to see so many companies becoming proactive to avoid “Success Disasters” – those opportunities when all the preparation, marketing, promotion and economy converge for your best possible outcome!

 

 

In the five years that SOASTA has been helping customers prepare for success, we’ve seen behavioral change in software testing driven by their end-user activity and affordable access to the resources needed to test earlier and faster, therefore more completely. The lower costs enabled by cloud computing, on a platform that enables anytime access and isolates issues instantly, have evolved best practices that yield results measurable in successes like this year’s holiday opener.

1)    Test early. Many of our customers began a few years back with “Hail Mary” types of tests in production systems sometimes a month or even weeks before Black Friday/Cyber Monday! Once they lived through that drill, they adopted cloud testing as part of iterative development and began ramping up much earlier to in the dev cycle.

2)    Test often. When there are no limits set by high license and hardware costs, testing can become a continuous activity. One customer – on the list mentioned above – had a goal of 68 tests to prepare for their big day. Without cost limits, they were able to run more than 1,100 tests!

3)    Test for more corner cases. When costs of hardware are minimized, the types of tests you can run are limitless. Testers in the past worked around scale limits or tweaked tests to make a 10-user license look like 100. It’s now an everyday practice to run tests of 100% , 200% or even 500%+ user load to stress and measure systems to their max capacity and beyond … quickly. This opens the door to many other test types to stress individual services, tiers, additional apps, and importantly, all the mobile load that could impact success.

4)    Test in Production. Once a taboo, every consumer-facing retailer we know is now testing in production. There is no other way to fine-tune the real prod infrastructure – from firewalls and load balancers to databases. Additionally, the latency effects of global distribution on end users – both web and mobile – can’t accurately be measured anywhere than in the “real” environments.

Affordability makes testing early and often a reality. From the people you hire to do the work, the hardware required to run realistic tests, the cost of the tools, to all the time users save in prep and setup, cost savings across all these areas has led to “Happy Holidays” for SOASTA users!

 

Guest Post: Continuous Integration for Mobile Apps with Jenkins: SOASTA CloudTest for iOS Apps

Posted by on

Mark Prichard is Java PaaS Evangelist for CloudBees. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was Product Manager for the WebLogic Platform. A graduate of St. John’s College, Cambridge and the Cambridge University Computer Laboratory, Mark works for CloudBees in Los Altos, California. Follow Mark on Twitter and via his blog Clouds, Bees and Blogs.

One of the most popular topics we’ve featured in recent webinars and meetups has been how to do fully automated, gesture-aware “touch testing” of mobile applications, using Jenkins running on CloudBees with SOASTA’s CloudTest Platform. This really is cloud computing squared with Jenkins in the Cloud for continuous delivery and SOASTA’s CloudTest providing touch testing for mobile apps as a cloud-based service. I promised a while back to do a detailed write-up on how this all works and here it is – there are also online examples and a video demo.

If you’ve been following my series of blogs on CI for mobile apps with Jenkins, you’ll be familiar with Stockfish by now: it’s a great open source chess application and I’m going to use the iOS version as a basis for this blog. The source code for the project is on GitHub and SOASTA have a very detailed tutorial that uses the same application, if you want to drill down into all the various testing scenarios that  CloudTest supports. I’d like to thank Mukul Sharma over at SOASTA, who did a great job puttting the original demo together.

To keep things simple, in the online example I have two jobs: stockfishchess-ios-install-touchtest and stockfishchess-ios-run-touchtest. The first job add support for touch testing, registers the app with the SOASTA CloudTest service, builds the iOS app and deploys it to a (tethered) iPhone/iPad device; the second runs the actual test. Note that tethering is only actually required to allow the first job to run fully automated: Apple only currently allow you to push a new app to a device without a manual OK if that device is tethered.  Once the app is installed on the device, tests can be run over wi-fi although I won’t cover that here.

The first job consists of a standard build fromGitHub plus the Jenkins Xcode plugin for the actual build, but the Xcode build is bracketed by two SOASTA utility commands that execute as shell commands. These utilities can be downloaded from the CloudTest Welcome page (look for the Downloads area on the right-hand side) and invoked locally on the MacOS slave, as in my example config. Alternatively, you could configure Jenkins to pull these down and unzip them on-demand, using curl and these links: MakeAppTouchTestable.zip and iOSAppInstaller.zip. This would actually be a good idea, since it minimizes the amount of configuration needed on the slaves and it ensures that the build is using the latest versions of the jar files, as SOASTA update these fairly frequently: I’ll be adding that to the online example configuration shortly.

The first adds some instrumentation support to the app to make it touch testable and registers the app version with the CloudTest service:

 

 

The second (which runs after the iOS Xcode plugin builds and packages the app) pushes the newly-built app version onto the tethered iOS device. As I explained in a previous blog, we use the Jenkins ${BUILD-NUMBER} environment variable to give a unique CFBundleVersion identifier, which can be related back to the the Jenkins job and the Git history. Finally, if the build and push succeed, in my example we call the next job in the pipeline to run the actual tests:

Here’s what you should see in the console output for the job (omitting the usual preamble and the xcode output): the first snippet shows the two files (Main and AppDelegate) that are modified to make the app touch testable and the registration of the app with CloudTest (I’m using the -overwriteapp flag to allow continuous builds without manual intervention):

 

 

The second snippet (below) shows the push to the iOS device:

 

For this example, I’ve used a shorter version of the “Fool’s Mate” test composition described in the SOASTA CloudTest tutorial. To record and run tests, you need to use SOASTA’s browser-based TouchTest Agent on the iOS device to connect to the CloudTest service. Here’s a screenshot of the CloudTest console (on the left, showing the Fool’s Mate composition, with a simple test for the Checkmate! dialog at the end) and the TouchTest Agent that runs on the device:

 

 

Now let’s look at the second Jenkins job, that actually runs the test composition. The configuration is simple: we execute a third downloadable SOASTA command line utility (scommand.zip) to connect to CloudTest and then publish the results in standard JUnit test format. A nice feature provided by the SOASTA CloudTest Jenkins plugin (also downloadable from the CloudTest Welcome page) is that test failures can link straight back to the relevant place in the CloudTest web console, so that you can see exactly where the test failed. Here’s the configuration and the output from running the job (remember that you need to have the TouchTest Agent on the device connected):

 

 

As well as being an incredibly useful way to do detailed, fully automated testing for mobile apps, it’s also a lot of fun to watch the tests executing on the device! Here are a few screenshots of the Fool’s Mate test composition plus the test results in Jenkins: of course, in real life you’d want to run whole suites of tests and include those with your unit test, code coverage and other reports, but this should give you a feel for what is possible:

 

 

Product Innovation. How SOASTA’s Changing the Game.

Posted by on

Infrastructure and services are major challenges faced by those in the automotive industry trying to drive the adoption of electric vehicles. Whether it’s charging stations to enable long trips or qualified mechanics to keep the car running, product innovations such as electric vehicles require complementary adaptations in the services offered to support them.

When we introduced cloud testing there were very few in the industry that knew much about the cloud … or testing at scale … or testing in production … or iterative testing leveraging real-time analytics. SOASTA’s team of Performance Engineers filled the gaps, first by providing on-demand load and performance testing for consumer facing websites and applications and eventually by sharing the resulting best practices and CloudTest Methodology with customers who have adopted CloudTest, such as Avaya, Intuit, Dillard’s, Best Buy, Old Navy, Hallmark and many more.

In developing TouchTest for mobile test automation, we’ve once again introduced an innovative product to a market in desperate need of a solution. The compression of the software development life cycle, the need to precisely validate the accuracy of multi-touch, gesture-based applications, the scale at which mobile apps are deployed, and the proliferation of device types means that it is NOT business as usual in the QA world. TouchTest is designed to address these issues with a unique approach that eliminates the need for emulation and optical recognition, does not require jailbreaking or tethering, and enables testing from inside the app that goes far beyond the UI level.

With the launch of our Mobile Services Division, we’re matching product innovation with services innovation. By helping customers build and manage mobile test labs, and combining best practices with a proven methodology, supplemented by accredited partners from around the world, SOASTA is once again filling the capability gap and offering the infrastructure and services needed to support innovation!

You’re Gonna Need a Bigger Boat

Posted by on

An Internet record was shattered the other day and few Westerners took notice.

On November 11, 2012 – known as “Singles Day” and “Double Sticks” – during their annual half-off sale, one company, Alibaba, who runs the Chinese equivalent of Amazon, TaoBao.com, took in over US $3 Billion (RMB 19B) in revenue.

This beats the total sales of ALL US retailers for Cyber Monday 2011 by $1.75B!

What kind of traffic is this? Well, a lot more mobile than in the past.

Here are some great stats:

  • 10 million visitors in the first 60 seconds
  • PC to mobile user ratio of 3:1 (up from 5:1 last year)
  • The first hour saw 7 million mobile users drive $16M (RMB 100M) in transactions
  • Total mobile transactions of $150M (RMB 940M)
  • 40% of China’s Internet users were active
  • Bank transaction flow exceeded six times usual traffic

Congratulations to Alibaba and Chinese vendors and consumers for such a record achievement! While there were reports of web slowdowns, shipping logistics issues and third-party service outages, executing on the biggest day of Internet revenue in history is a massive and landmark accomplishment.

What this says for large scale mobile and web site preparedness prompted my blog title – that famous line from Jaws. If you’re planning to continue to deliver business services to customers via mobile and web applications, you’re going to need to prepare for the inevitable – wild success! And not only will you need a bigger “boat”, you need to know in advance that the boat won’t sink.

At SOASTA, we built for scale in anticipation of the literal explosion of the mobile Internet. We harnessed the cloud early, as we knew that a major obstacle to testing at scale was ready access to hardware to simulate large numbers of users. Cloud was the perfect answer and a good choice for us. As the cloud has grown, so have we. In a tweet today, Tom, our CEO, reflected back to 2008 when Amazon Web Services had 1 cloud datacenter. Now they have 60, with the latest opening yesterday in Australia.

This Global Test Cloud is the “bigger boat” for mobile and web performance testing.  In addition, our ability to quickly dig into performance data while tests run assures that time is used wisely to fix issues instead of chasing their source long after tests conclude.  With access to practically limitless scale for testing and technologies build for the big data collected during large tests, compelling tests can be quickly deployed to simulate any real-world condition. For example:

  • Study and establish user experience baselines with real traffic from multiple geographies
  • Hit peak and beyond-peak load levels with real mobile and web traffic
  • Measure mobile user experience and hammer sites with a mix of mobile and web business scenarios from buying gifts to trading stocks
  • Tune your end-to-end infrastructure and third-party services for peak performance under stress conditions

The consumers of China have shown us the future is now. Regardless of how you test, it’s critical you do, and for those issues you will only find under extreme conditions.

If you don’t, the sharks may find you.

 

Sources:

 

We are a society obsessed with speed.

Posted by on

The fastest car on the planet, the Thrust SSC, broke the sound barrier and holds the land speed record at 763 mph. Last week, the biggest rocket fired in the UK in 20 years was a test for the one that will propel the Bloodhound SSC, the car that hopes to crush the record and cross the 1000 mph threshold … on wheels.

 

 

Fast or faster isn’t good enough. Fastest is the goal.

As technology consumers, our expectations for the apps we use – from shopping or viewing videos to trading stocks and updating inventories – are similar to the pursuit of land speed records. What was once fast is redefined with each new generation in technology. And each new generation of technology arrives quicker than the last. For organizations tasked with testing the performance of web and mobile systems at the scale of modern consumer web and mobile usage, keeping up can be a constant pursuit.

At SOASTA, we’ve found that our customers have redefined speed – and apply it to every area of their test process to keep up with the pace of technology. To achieve speed in testing requires process compression before, during and after testing.

Applying Process Compression

Test creation: Traditionally, test creation requires deliberation about what to test, then a lengthy period of test development using code-based approaches and complex test debugging and editing. Process compression in this phase can eliminate 10-90% of the pre-test prep and get teams to the act of testing more quickly.

Test Hardware Preparation: Once tests are prepared, to run a performance test at scale requires servers. A large test – e.g. one for thousands or millions of expected users – requires a lot of servers, more than most have on hand. Procuring (buying – begging – borrowing – asking IT) takes time. So does set up and administration. Tests must be run at full scale to test full scale so looking for ways to eliminate the delays in test hardware provisioning will cut out delays (and costs) in this phase.

Analysis: We test to make determinations about pass, fail, go and no-go. It is that simple. The faster we can get to the answers, the better. What’s surprising is how long we sometimes take after tests to determine if the result was good or bad, and what to do in the case of bad. There are even companies that help other companies analyze the results of the tests that took too long to run in the first place! The goal of testing needs to be an eternal pursuit of actionable results, faster. Applying techniques that enable analysis and even issue resolution during tests is proving to be the best way to cut these cycle times.

At SOASTA, process compression is a strategy. The result is a set of solutions dependent on a visual approach to testing that eliminates complexity but delivers power; is cloud-based to eliminate server provisioning at unlimited scale; and is propelled by a big-data analysis engine that runs faster than the Bloodhound SSC to deliver test results instantaneously.

In a society obsessed with speed, technologists don’t have a choice about keeping up. We do have a choice to pilot the rocket car or try to catch it as it screams by at 1000 mph.

Baseball, Hot Dogs and Start-Ups

Posted by on

Once upon a time I played a little baseball. Despite this rather innocuous beginning to my career, for the past 35 years I have been building startup software companies in The Valley. The surprising part is just how similar the two experiences are, especially around success and failure.

Hear me out.

Growing up in the Bay Area made it quite logical for one of my childhood baseball heroes to be Willie Mays. Willie once said, “In order to excel, you must be completely dedicated to your chosen sport. You must also be prepared to work hard and be willing to accept constructive criticism. Without 100% dedication, you won’t be able to do this.” Find me one accomplished Silicon Valley entrepreneur that isn’t just as dedicated to their idea and has not endured countless hours of criticism (primarily from VC’s) yet they have still prevailed. My point is this: the attributes that make someone successful are shared whether you are an entrepreneur, a baseball player or in any other field for that matter. You must be passionate and you must be “all in”!

 

 

Another significant similarity between baseball and The Valley is the value of a team. For years I have watched teams made up of some truly amazing athletes never win a championship. Even Willie Mays, arguably the greatest baseball player ever, won only one championship and that was at the beginning of his career. Ted Williams, Willie McCovey and Ken Griffey Jr., other gifted athletes, never won championships. Yet in 2010, a hapless group of baseball castaways with a few extremely talented pitchers pulled it all together to win Baseball’s Championship for the San Francisco Giants.

What made that group different then the more talented teams that failed before them? It’s the team. A group of people that share a responsibility toward achieving a common goal is greater then a single person with a personal agenda every time. And that’s true in the Silicon Valley as well. Sales people generally do not write code. Engineers generally don’t sell. And none of these people want to count revenue and or market the products. It takes a village to be successful in The Valley, and the more focused each person is in achieving the common goal, the greater the success.

Baseball and The Valley have defined my life. Passion, dedication and team have made my life rewarding so far.

Now let’s play ball!

Digging into the RUM at SOASTA

Posted by on

Real User Measurement (RUM) is a new analytics tool for marketing and operations professionals. Marketing has a huge data component and good marketing is all about how you use that analytical data.

A month ago we installed our new product, mPulse, on SOASTA.com. mPulse is a product we co-developed with Lognormal, the leading Real-time User Measurement company that we just acquired. We’re getting all kinds of great data that allows us to correlate page load time with user behavior and user engagement. More about this in a moment.

Think of RUM as a car dashboard. The 1972 wood-paneled Ford Country Squire station wagon you spent your childhood in had just a few instruments to display: speedometer, fuel gauge, oil temp. Forty years later the Toyota Prius has digital instrumentation that includes a speedometer, fuel gauge, odometer, instant fuel economy, shift position indicator, fuel consumption history, average fuel economy, distance to empty, average speed and trip distance … all delivered in real-time. To say that we have access to much more data now is a gross understatement.

RUM gives us the same data progression but it’s for our web site and apps. As Google Analytics users, we have good data on visits, time on site and overall bounce rates, but lacked real-time measurement on what was behind those stats. We needed to better understand how our web site performance, page load times, connection speeds and geography affected our user’s experience. Now we get much richer data sets and can drill down and look at performance issues based on page groups, browsers, geography or the available bandwidth a user has to access our site. This allows us to make data-based conclusions about how web performance effects time on site, page views and bounce rates.

 

 

For example, understanding the number of web visitors that come to our site using a Chrome browser vs. IE 9 and the average page load times of each allows us to prioritize our optimization efforts for each browser type. We also know how fast page load times are in the US, South Africa and in all the other countries that bring visitors to SOASTA.com. This helps us determine our content distribution and caching strategies for the geographies we serve.

All of this is critically important because research from Aberdeen Group tells us that that 25% users abandon a web site after just three seconds.* That can translate into a whole lot frustrated users and abandoned shopping carts.

This sort of data helps us, and people like you, figure out the right balance for different types of content. Maybe your home page needs to have a fast load time but your product pages, which are heavy with multiple images, may not because people are willing to wait little longer for content they are especially interested in. You can use this data to reduce bounce rates and provide relevant content in a format your users want.

Imagine this as a continuous cycle of user experience testing. Smart businesses use real-time data to drive better users experiences, continuously testing, deploying and measuring those experiences. Rinse and repeat. It helps you provide the online experience your customers want and your competitive business demands across all your properties: web sites, mobile devices and native apps. And as a marketer, I think that’s cool.

*Aberdeen Group, First Class Mobile Application Performance Management, Jim Rapoza, Aug 2012

Peter Galvin, SOASTA SVP of Marketing, authored this post.

 

 

Delivering Contextual Intelligence to Mobile & Web Developers

Posted by on

The enterprise mobile world is in its very early days, with most companies still developing their strategies (native, HTML5 or Hybrid).  The fact is, and even for the most advanced mobile developers, most have been working in the dark when it comes to understanding how users interact with their applications. This lack of visibility into the mobile user experience is becoming more and more of an issue as they attempt to prioritize features and functions that will ultimately show up on increasingly smaller and diverse screens.

An app’s success is most often dependent the developer’s ability to grock the subtle differences in how the end user wants those features delivered. Simply stated, mobile marketers and developers lack user-related context.

Without any form of real context, web and mobile developers are taking a stab in that dark as to what and where features should be displayed. The first generation monitoring (RUM) tools focused almost exclusively on web applications and provided only a limited amount of intelligence derived from correlated information. These tools failed to capture any significant data on the mobile user experience.

The contextual differences between web and mobile are the key to delivering the best user experience in both formats. Fortunately, that gap in knowledge ends today with our announcement of SOASTA mPulse.

mPulse delivers the necessary intelligence back to the professionals who need it most. Its real-time analytics dashboard captures not only the actual web user experience but all forms of mobile user experience. Without context, developers, testers and marketing are doing a lot of guesswork. And in this New Mobile World that’s not enough.

Get involved with SOASTA CloudTest Mobile Development!

Posted by on

Back in February of this year we introduced a brand new capability to CloudTest allowing developers and testers around the world to perform functional test automation for mobile devices. We also introduced our Early Adopter Program, allowing participants early access to new product features before the general release of the product. We received great feedback from our users and since then our engineering team has worked tirelessly to integrate this feedback into our product.

A couple of weeks ago we announced the latest major release of CloudTest Mobile which brings to market unprecedented features, allowing you to deliver even better quality mobile apps. As with our initial release, we’re opening up an Early Adopter Program dedicated to you developers and testers! Being part of this new program will allow you to:

  • Get a dedicated and self-service environment for your developers and testers to try out our product on one of your project
  • Get a direct relationship with our engineering team for support and discussion
  • Be the first to get the latest features directly available to you on your environment
  • Provide feedback and enhancement suggestions to our product management team (That’s me! How cool is that!?)

Today, we are signing up participants currently developing and testing mobile apps for Android and iOS.

I’m very excited to manage this program and already involved with a lot of people testing some amazing apps! I would be delighted to get you onboard and give you access to a preferred environment for faster, more precise testing of your mobile app. If you’re interested, let’s have a chat and decide how to move forward. I’ll be glad to work directly with you or maybe point you to our free product CloudTest Lite which allow you to test your mobile apps against one device (either iOS or Android).

You can contact me by email at fberinger [at] soasta.com.

What’s new in CloudTest Mobile

Additional support for Android, mobile web and iOS hybrid apps

  • In addition to iOS, precisely record and replay on all major mobile OS including Android Jelly Bean without rooting your device
  • Integrate TouchTest Technology with your Android or hybrid apps faster with our enhanced integration utility
  • Full support for mobile web apps
  • Complete support for embedded UIWebView in your native Android or iOS app
  • Execute your functional tests on any number of Android devices across different version of the OS

Expand the scope of your functional testing with screenshot validation

SOASTAimagevalidation Get involved with SOASTA CloudTest Mobile Development!

  • Add screenshot validation automatically for all your actions or specific actions during recording. Screenshots will be used as baseline for validation during the execution of your functional tests
  • Use an existing screenshot from a test results as a new screenshot validation
  • Add comparison tolerance to avoid false positive
  • Mix and match screenshot validation with our powerful object-based validation methods

Monitor your device key performance metrics during the execution of your tests

soastamonitoring1 Get involved with SOASTA CloudTest Mobile Development! soastamonitoring2 Get involved with SOASTA CloudTest Mobile Development!
  • During test execution, get access to your device CPU, memory and battery utilization
  • Understand the data consumption of your app in real time
  • Compare performance measurement across any number of devices, carriers and location

New Object Locator

soastaobjectlocator11 Get involved with SOASTA CloudTest Mobile Development!soastaobjectlocator2 Get involved with SOASTA CloudTest Mobile Development!

  • Our new object locator capability allows you to find and capture any object of your app during the recording of your test on a real device
  • Use the identified objects for your validations, waits or outputs

New OpenGL support

  • Full support for all OpenGL apps and games as well as all gaming framework based on OpenGL (such as Coco2D)
  • Record and get access to any OpenGL variables and methods to create your tests

Access all variables from your application to build stronger tests

  • New AppInternalValue method allows you to expose all variables and methods from your application code and use them inside your tests as validations, waits and outputs

Continuous Deployment for Mobile Apps with Jenkins: Back-end Services and Integration Testing

Posted by on

In last Tuesday’s post Java PaaS Evangelist Mark Prichard described how to configure Jenkins to manage Xcode builds for iOS applications using MacOS slaves, which he demoed at the recent Silicon Valley Cloud Computing Group Meetup. For this demo, he built a simple servlet-based chess server with REST web services to record games and moves, using JAX-RS, JAX-B and a MongoDB backend. You can find the source code here on GitHub. Click here for more info.