Should You Run A Beta Test Yourself?

April 9, 2009 by Kohan Ikin

For more news like this:

I was recently asked by a reader of my article "How To Run A Beta Test... Or Not?", why didn't I use a professional beta testing service to avoid the problems I encountered? Good question!

I've never tried outsourcing beta testing to a dedicated service, so perhaps there's something I've missed out on. But there are several advantages to beta testing directly with end-users / customers that I don't think a beta testing service can replicate.

Before I continue, a caveat: this is written from the perspective of a small software startup developing consumer software - a microISV with minimal funding. It may not apply for a large software company writing bespoke or B2B software.

Testing on diverse platforms and real world systems

Obviously, we need to test our software on every version of Windows, and any decent beta service will offer this. There may be some doubts about whether they have older systems - eg Windows 98 may be obsolete, but a small percentage of our customers still use it, so we still do some testing on it. I assume that a beta testing service would be able to do that for us too.

Testing on multiple Windows versions is something we already do in-house though. With an MSDN Operating Systems subscription and a copy of Virtual PC, Parallels Desktop or VMware Fusion, it's easy to test your software yourself on multiple versions of Windows - even if you only have one PC. Honestly, this is something any serious developer should be doing already. (And we actually test all the way back to Windows 95a - you'd be surprised at the bugs that can be revealed by testing on those old systems!)

But the one thing developers can't test on - and that I don't think beta services can either - is a real world customer system. Some customers have software installed that conflicts with each other, has esoteric settings, or might even be junked up with viruses. By running a beta test with our customers, we see how our software works on systems that real customers will be using - our clean VM installations of Windows aren't accurate representations of that.

Would a beta testing service have all the software you need?

Another issue (for our software, at least) is whether professional testing companies have access to all the necessary software. In our case, that would mean every version of Photoshop from version 6 through to CS4, every version of Paint Shop Pro from version 5 to X2, versions of Photoimpact from v8 onwards, copies of Photoshop Elements, and so on.

We have access to many of those programs through our relationships with those companies. For some programs that we don't have, our customers use them and can do some testing for us and provide feedback. But for a professional service to buy those licences would cost several thousand dollars, and some of those programs are no longer available for purchase anyway (except, perhaps, secondhand via eBay).

This is probably something unique to our software, though. If your program is a more mainstream out-of-the-box application, it might not be a concern for you.

Customer beta tests help you learn about your market

As part of our beta-testing signup process, we ask for a lot of information. We do this to screen for serious/genuine applicants, and to make sure we get a wide cross-section of users when testing (eg making sure we have people of all ages and genders in our beta testing). But from the signup applicants, we also get a sense of who is interested in our product - we might see more of one age group or gender than another. By talking with our customers during the test, we learn more about their interests, the websites they visit, and their needs for the software. We can also see from the number of people signing up whether a product is likely to be popular or not. If people aren't signing up for the beta, then maybe we need to tweak our marketing messages or reconsider the project.

Beta testing with customers helps you launch your product

On one occasion when we worked with our beta testers, not only did they get a free copy, they were also given a limited-time discount coupon to give to friends. That created significant word of mouth and was a key factor in the launch of the product. While you can approach bloggers to talk about your product, people are more likely to talk about your product if they're a part of its creation. And our beta testers should feel that way, because the testing is a significant part of our development & design process. We try to credit all our testers in the software, as a recognition of that.

Big companies don't use beta testing services (or do they?)

Many indie developers have been involved in private beta testing for large commercial software companies before. We know that they work directly with their customers for beta testing purposes. Perhaps they're working with beta services as well and we just don't know - in fact, they probably are. But if that's the case, and the private beta testers are getting the software after that outside testing, then the bugginess of the software we received doesn't say much for those services!

In all honesty, I don't know this for certain. What I do know is that amongst smaller startup developers, it's rare to hear of them outsourcing to a beta testing service. Of the larger companies I've worked with, it was never disclosed to me if an outside group was also involved, but that's understandable.

Can a startup / microISV afford a beta testing service?

This is probably the biggest issue in the way of a startup software company using a professional beta testing service.

Many of the beta testing services that I've seen provide no pricing on their website. While that's totally understandable, there's a general rule of thumb that startups follow - if the price isn't listed, you probably can't afford it. An indication of costs (without having to "call for pricing") would help startups / microISVs make that decision early on. Without that info, they'll often just ignore the business.

Anyway - none of this is meant to be negative against professional beta testing services. I've never used outsourced testing services personally, and maybe they're quite useful. If you're having difficulty finding beta testers for your product, perhaps it's an ideal service for you. But I don't think an outsourced beta testing service can completely replace the benefits of beta testing directly with customers.