SOASTA just announced that Web UI/Ajax Testing is now shipping as part of SOASTA Concerto. We now have a complete, totally integrated, professional solution for testing Web 2.0 applications. Every layer of the web application. Every testing type. As I have discussed before, all enterprise software is on a 10 year journey to be rewritten for the Web. As I discussed here, it is vital that developers and testers gain native tools for testing on the Web. While we have much more to contribute as we continue to develop our products, I am incredibly proud of what we have delivered today and of the team that built it. Thanks.

In my last post, “Web Testing Is Network Testing,” I began a conversation. Modern Web applications are difficult to test because there are so many events occurring at many levels. Application behavior is effected by packet events that can be extremely subtle. So, how do you know “what’s going on”? Sometimes it can be extremely difficult to tell, but it is only possible if you can record and analyze the message and packet traffic in the Web application.

Here is an example from Susan Crawford’s blog and the original article in The New York Times:

http://scrawford.net/blog/comcast-is-pretending-to-be-you/1031/

http://www.nytimes.com/aponline/technology/AP-Comcast-Data-Discrimination-Tests.html

The article is about traffic shaping by Comcast. Comcast is allegedly using technology that recognizes BitTorrent downloads and sets the RST flag in the TCP header:

Each header in a TCP-labeled packet (think front of an envelope) has a number of fields. One of these fields includes “flags” that are applied to the packet. One of these flags is called RST for “reset the connection.” The Comcast system . . . was setting the RST flag for both sides of any communication that it believed (probably through traffic analysis) was using BitTorrent.

There will be a huge debate over whether this kind of “traffic shaping” is desirable. That debate will rage on for a while.

For my purposes this is an example of the underlying complexity of network applications. The packet traffic (and therefore the behavior) of an application can change based on events that occur while packets move across your infrastructure through routers, load balancers, network accelerators, Web servers, application accelerators and application servers. In the Comcast case, traffic analysis recognizes that a BitTorrent download has started and Reset Connection flags are inserted into the network traffic in each direction. This action is invisible to the end user and is only visible if a network analyzer is applied to both ends of the traffic. Similar technology is used for load balancing, application acceleration (caching), and for security applications.

The behavior and performance of this entire infrastructure is interdependent. GUI testing approaches will not get you there. Web Test Is Network Testing!

Two weeks ago I gave a speech at AjaxWorld West 2007. My session was about best practices in Ajax testing. There is a theme that runs through all of the work SOASTA is doing to build our products: “Web Testing Is Network Testing”. It is not possible in this generation to test the UI only and expect to cover all of your bases. N-tier Web applications are way too complex for UI testing to be all that you need. I will write more deeply about this in the future but here are some things to think about:

• Modern Web applications are complex composite applications that will include a browser UI, a Web server, a Web application server and a database server. All of these components have hundreds if not thousands of parameters that can dramatically affect performance
• There are multiple simultaneous message streams flowing between these components (SOAP, http, Ajax, JSON, REST. . .) each with its own subtle behaviors
• Rich Internet Applications deliver dynamic UI experiences by using messaging to deliver asynchronous events and content
• Messaging events and browser events make up the event stream that the UI tester has to validate

This is a complex architecture that is more complex to test. In order to build a test the test designer now needs visibility into and an understanding of the message streams that are flowing in the application. We don’t think that GUI testing tools do the job here and we are intent on fulfilling this need.

Last week SOASTA announced the sBox Load Testing Appliance. Our idea, from the beginning is that:
1. All enterprise applications will move to the Web
2. Therefore, all enterprise testing will become Web testing
3. The market needs new players with new technology to address Web testing
4. Load testing on the Web is more important than in the client/server world because you are on the Web directly serving your end customers
5. Load testing is too hard
6. Load testing has been very expensive
7. Many companies do not load test because it is too hard and too expensive
sBox is preloaded with SOASTA Concerto (our very fine Web Testing application) and a single instance of sBox will support up to 2,000 Virtual Users for $2,000/month or $1 per virtual user for dedicated, 7×24x365 load testing.

The early response has been very encouraging. The value proposition is pretty easy to understand. Complete, professional, affordable load testing.

What’s Next?

Our intention is to support all types of Web testing (load, stress, functional, Web UI/Ajax/Flex/SilverLight . . . whatever) across browsers (Safari, Firefox, Internet Explorer . . . ). We will have another release with those capabilities coming up next. We are on a march to become your Web testing vendor of choice.

Give us a look. I think you will like what you see.

8:30 a.m. Waiting for the conference to start here at Google’s New York office. I flew across the country yesterday with Jason Huggins (inventor of Selenium). Wonderful person whose creative spark has already forever changed how UI testing will be done. I am really looking forward to spending the next 48 hours with the world’s best testers. Test automation is such an important topic for the Web today. Strategic for Google and many others, but Google is the company who stepped up to sponsor this conference.

Here is a link to the GTAC Community Blog

9:25 a.m. Allen Hutchison from Google’s London Office is kicking off right now. He states, “GTAC started last year. The conference is centered on the practice and discipline of test engineering. There was an overwhelmingly positive response to last year’s conference. So here we are again”.

9:39 a.m. Allen kicked off the conference by discussing the four principles for GTAC:

Conversation
–Keep it small
–Leave ample time for questions
–Single track so that everyone has the same context
–Lightning Talks

Community
–Promote Testing as an engineering discipline
–Practitioners and academics

Technology
–Automated Testing are the most important Tags

Innovation
–Lots of new ideas

9:53 a.m. Keynote: Pat Copland, Test Engineering Director for Google.

Topic: Google Automation Testing

What is it about Google’s culture that works?

$60B is lost each year due to software bugs.

Google’s testing culture emphasizes People + Science + Improve. Give Tools not the answer. Use technology to solve problems. Measure the approach and the product itself. Google evangelizes testing internally with a lot of effort. Emphasis on Developer Testing. Continuous build systems. Early problem detection coupled with continuous improvement. It is very impressive to see what they are doing.

One of the most important ideas Pat Copland speaks about is the goal that a product can be certified and released within 1 day of automated testing. The engineering teams need to take test automation very seriously in order to hit this goal. Great, great example of leading edge test engineering. Continuous improvement of the test automation is an idea whose time has come. I recommend every software engineer and test engineer watch the video of this talk.

I am #28 in line at the Apple Store in Stanford Shopping Center. I have been here since 10:00 AM this morning. Why come here? Because Stanford did not allow people to camp out overnight so I thought I had a good chance here. That turned out to be right. We will see.

Why am I doing this?

Because this is a revolutionary device that will give all of us the Internet in our pockets. If you are in the software business then you need to understand what this will mean. At the simplest level, the number of devices with full browser capability is about to explode. A lot more phones are sold every year than PCs. Cross-browser (Safari, mobile, Opera, …) compatibility just got another boost in priority.

Because I love this stuff! I have had a lot of fun today in line with well over a hundred other geeks (mostly but not all men). Apple brought out water, Starbucks gave away samples, lots of camaraderie, lots of fun.

I just synched my Treo for the last time so I can transfer contacts. Good bye old friend. A brilliant piece of work at a point in time but Apple is pushing things forward with design innovation that leaves everyone else in the dust. Cannot wait to get my hands on the device.

2 Hours left. In Line To Get an iPhone. : )

Steve Jobs announced the other day that Apple will allow third party applications on the iPhone via Safari + Ajax. Real Internet applications delivered in the browser on a mobile phone. This is a continuation and confirmation of a hot trend – all enterprise applications are moving to the Internet. Enterprise applications will be available in your pocket everywhere you can get mobile service. The same experience, the same application on your phone. This is incredibly important. Within a year, every smartphone will have a first class browser capable of running Ajax applications over the Internet.

Steve also announced Safari for Windows. So you might not have to buy a Mac to test Safari compatibility. We will see. So far, Apple has gotten every decision right from the market’s perspective. Microsoft is still learning this.

What does this mean for the Web Tester?
1. Your end users will be using Safari. Not testing Safari yet? You will be.
2. Your end users will be using other browsers that support Ajax. If memory serves, there are ~87 million smartphones sold every year. The people purchasing them are your employees, your partners and your customers. As first class browsers get added to these platforms in order to compete with Apple, the list of browsers you need to test will expand. Cross browser Ajax compatibility will become very important.
3. Ajax UI testing will become more important.
4. And subsequently, Ajax Load testing will become important.

An underlying trend to all of this is that Web applications are being built to support far more end users than ever before. A generation ago, applications were built to support thousands of users. In this generation, that may work on a normal day. But what happens when your Marketing Department runs a very successful ad that brings 10 or 100x more users than normal. If you are lucky it means a crash project to load test and deploy more hardware. If you are unlucky, then . . .

At SOASTA, we do all of our development using J2EE, on Macs (Robert Scoble wrote about this a while back). One of the reasons we do this is for portability — we can write our code on a Mac and run it on a Linux server.Recently we were surprised by a platform specific-performance issue.We have started load testing our product using our product. How much load can we generate on how much hardware is a key issue for our market and therefore this is really important for SOASTA’s sales and marketing team. While doing this testing we were very surprised to find that one of our machines was getting way better – three times better — performance than any of the other machines. My MacPro workstation was able to generate three times more SOAP calls than the other machines in our lab. This was a puzzle because one of these machines was a brand new 8 way MacPro with 8 Gs DRAM. It got even more puzzling when my MacBook Pro laptop produced the same result. Something was causing a bottleneck that made extra CPUs and memory not matter. Our entire Server team stopped what they were working on to focus on solving the puzzle.

We spent a long time working on this. We even made a visit to the Apple Store to see if one of their resident Geniuses could come up with anything. I won’t hold you in suspense, though. The difference turned out to be that both my machines had the SOAP server in their “hosts” file. For those of you who don’t know, the “hosts” file is used by Unix and Windows operating systems to do a fast lookup of the IP address that belongs to a certain host name. For example, you can put in the line:

208.64.59.60 www.soasta.com

and the next time you browse to the SOASTA website your machine would not need to contact a DNS server to get the IP address.

Now, this could almost explain the difference in speed, except we were using the IP address to talk to the server! So clearly there was no reason for the hosts file to be involved.

It turned out that this was a bug in Java: bug 5092063*, to be exact.

It was introduced in Java 1.5 and caused the Socket.connect() method to make a call to the name server when none was necessary. It also turns out this bug is already fixed in the latest distributions of Java 5, except for the version that’s available from Apple for Mac OS X.The lesson learned is that even though you think you’re running platform independent code, platform specific configuration issues will have a big effect on your performance.The only way to know how you perform is to load test continuously. Seemingly minor parameter settings can negatively impact the throughput of your web application in ways that are always a surprise. P.S. We now have a workaround included in the product, so we get great performance regardless of whether your version of Java is fixed or not.

* To be fair, this is really a bug in the Sun implementation of Java, not Java per se.

Load testing has been in the news quite a bit over the last few weeks as Intuit and RIM (owners of the very popular Blackberry device) made headlines with performance problems that they both blamed on inadequate testing.

Intuit acknowledged that “ . . . hundreds of thousands of TurboTax customers were unable to file their state and federal taxes before the Tuesday midnight deadline because of a slowdown of the company’s computer servers.” Here is a link to CNNMoney, “Late filers swamp TurboTax”

Brenda Michelson’s blog entry about RIM’s system outage, “Blackberry Jam: Poor Systems Testing” is here.

Ouch! is exactly right. Both companies were front page on all of the national newspapers.

We will see more of this kind of news and here’s why. Enterprises are not testing their systems to anything like the level of quality and coverage that is needed today. The architecture of enterprise software has permanently changed. The Internet has become the primary delivery channel for enterprise and consumer applications and the volume of users and the complexity of these systems requires a lot more testing—and it isn’t happening. At least not yet.

Ten years ago, applications for customer self-service (things like banking, filing taxes, ordering products, reporting problems) were delivered through call centers. So, the application challenge was to build an application to support the thousands of call center representatives. Response times were considered “pretty good” if the average response was less than 20 seconds. All of us are being trained by the best sites on the Internet to expect far better response than that. Four second or less response times are now expected.

What a different world today!

In an excerpt from CNNMoney, “During peak demand, Intuit was processing 50 to 60 returns per second, he said. “We started seeing a very significant slowdown in the database,” he added, noting that the company processed more than a million tax returns Tuesday.”

Here in an blog entry entitled, “Intuit Overwhelmed, Blames Customers”, Brian Cantoni comments on his experience.

So the results are in. Your customers want to use the Internet as the primary connection between your company and them. They vastly prefer the Internet self-service model to any other, and they are expecting you to serve them well through your Web applications. The numbers are staggering (ask Intuit) and are going to get much bigger every year. The win for the enterprise is that it is far less expensive to support your customers over the Internet than any other way. The win for the customer is that they do not have to talk to anyone to get what they want. But only if you can serve them and expectations about performance, user experience and quality are extremely high.

The challenge to IT is very real.

Support hundreds of thousands of real end users instead of thousands. Deliver applications with 4 second or less browser response times even at peak. These are your real customers—not employees, and they will vote with their business if they are not satisfied with the user experience delivered to them over the Web.
Do it with a more complex system architecture than the really stable client/server architecture you have. More points of failure, higher volumes, less sophisticated and less patient users.

Do it with the headcount you have or with less. Deliver and the savings to your enterprise are immense. Fail to deliver and make the front page of the Wall Street Journal. As someone commented to me recently, “Murphy is alive and well … and he never sleeps!”

This is all very exciting.

Last Tuesday, SOASTA announced plans to extend SOASTA Concerto to support Web UI/ Ajax testing. When this capability is delivered in Q3 of this year, SOASTA Concerto will support automated testing of every important Web protocol (SOAP, REST in many variations, XML-RPC, JSON, HTTP, messaging middleware and Web UI/Ajax).

Why Are We Doing This?

Our strategy from the beginning of the company has been to provide an automated environment for testing all the components of a modern Web application that runs across HTTP. As I have discussed here and here , Web testing is hard because of the complexity of the technology stack used to deliver the applications and the diversity of implementation choices that developers can make. As a result, automated testing of Web applications has required very experienced, very technically savvy resources to build the testing. Since we never have enough of these resources, Web testing is currently not automated or is minimally automated. When we speak to practitioners the primary tools being used to drive testing are hand built scripts (Perl, Python, Ruby, …) and hand edited XML documents. Web UIs are tested manually or are being “tested by the users”.

Alternatively, we have met development teams that have invested a lot of money and resources building extensive test harnesses using C# or Java or C++. Lacking viable solutions to testing, developers who have to deliver will build their own tools. This is a very, very expensive approach. Companies we have spoken to are approaching a 1 developer to 1 test developer ratio and are currently spending millions annually building a proprietary testing environment. As one very insightful QA leader told us, “Until you try to test one of these Web applications, you have no idea how hard it is”.

This approach does not scale as we get serious about rebuilding enterprise software on the Web. Our interaction with the market confirms that we all need an easier approach that handles the detailed complexity and allows application analysts, consultants and end users to reenter the test creation process. The person who understands the WSDL and XML of the application will probably not be the person who can construct the Real World scenarios that will really be productive for your company.

Why Is This Important?

After many years of trials and standards development, rich Web UIs based on Ajax are appearing everywhere. Ajax based UIs combine messaging events and browser actions to deliver an application across the Web that combines the best of the Web and the desktop. End users get a great rich end user experience while the IT organization gets the ability to support a single application image from the server. Every leading software application vendor is moving rapidly to deliver new versions of their products to enable this approach. Customer-facing applications that provide self-service ability to your customers seem to be the market force pushing these new applications forward. It is a big win-win. Your company saves money because the customers support themselves and the customers are much happier with the service they receive . . . as long as the applications work and perform. Continuous, automated testing becomes very important.

How Can You Get Involved?

We are interested in lining up beta users and hearing about your unique requirements. Please go to http://www.soasta.com/ for more information.