Once upon a time, with a client far, far, away, I ran the QA Lab (when I wasn't testing) for a company that among other things, sold ISP services. Since this company derived a good amount of revenue from dial-up services, we had to have an array of setups that simulated the potential environments of the customer base. That meant boxes for every operating system from Windows 95 thru Windows XP. Before you ask, yes the environment included a box running the much-maligned Windows Me (no truth to the rumor that "Me" stood for Maligned Endlessly).
Oh yes, there was an Apple box (a beautiful G2), and a couple of boxes that ran VMware. I had to set up and maintain 20 boxes. Ever loaded and debugged device drivers for ONE machine? Now try doing it for at least a dozen of them. I always thought then that it was a special form of hazing.
The moral to the above is this: there is a financial and psychic cost to maintaining a QA environment. There is an extra cost if the environment is especially fluid. Infrastructure costs, labor costs to maintain, and more costs to adapt existing hardware. As a result, some are looking to the cloud to provide answers.
For a company that does not have the money or physical space to maintain a full-blown QA environment, and here I'm taking specifically about Web performance load testing, there are alternatives that may be a good fit. Let's take a look at 3.One solution is CloudTest. This application touts itself as load-testing solution that combines performance and load testing leveraging Amazon's EC2 environment to be used as an on-demand service. That last bit is important. You have to be certain you understand how the EC2 environment is impacting the load testing. As I mentioned in my last post, there are peculiarities of cloud service providers that need be carefully examined or you maybe left in the dark when problems arise.
Another is crowd sourcing as opposed to outsourcing. This somewhat controversial approach is the engine driving uTest. This is an application that calls upon a group of professional testers to test the performance for your Web application. There is a real issue with repeatability and subjectivity of issues found. As many battles as I've had in my life over whether a bug is severe or minor - or even a bug to begin with - well, I wouldn't want my paycheck to ride on that.
Keynote attacks the issue from a couple of different angles. LoadPro is a variant of outsourcing - and a better one I might add. You get the benefit of working with expert consultants but also retain control of the what, when and from where issues. And rather than soliciting strangers (insert your own punch line here), Keynote generates real load from its dedicated network (and from outside your firewall) and performs a complete load testing analysis. The advantage over a typical crowd-sourcing approach is clear - quality of testing results can vary wildly in the crowd-source (or as a friend of mine put it "mob-source") approach.
The other important component is that Keynote not only handles the testing, but manages the testing environment. And unlike a cloud service provider approach, you are not dependent on the vagaries of how a cloud service provider has implemented its network - for legitimate reasons to be sure, but not beneficial for real world load testing. Keynote's load generation is directly attached, if you will, to the Web just like your customers are.
Do you want to see what happens to your site when one million (1,000,000) concurrent users are hitting it? LoadPro allows you to do that, and at this time seems to be the only product on the market that does. Having scalable severs that you are not responsible for comes in very handy when there is a need to test asynchronous demand. Foxsports.com uses LoadPro to successfully load test their Web site as does BestBuy.
Among Keynote's other tools is KITE (Keynote Internet Testing Environment) which allows a tester to create their own path through a Web site and then run a performance test in the cloud simulating the performance from cities around globe - great if your business has global reach. And you do not need to host your own internal server farm. In a couple of previous columns, I've made use of KITE to get a sense of website performance on various sites. Cost of server usage? Zero.
In nutshell it's the question that was posed to me last week: "How intensive are the applications you are testing and how reliable is the service?" Before selecting your Web performance load testing solution do your diligence so you won't get caught flat-footed when your Web site goes down.
Comments