This is the eighth post in my Customer Research series.
What is a beta program? The Wikipedia has the following definition:
“Beta” is a nickname for software which has passed the alpha testing stage of development and has been released to users for software testing before its official release. It is the prototype of the software that is released to the public. Beta testing allows the software to undergo usability testing with users who provide feedback, so that any malfunctions these users find in the software can be reported to the developers and fixed. Beta software can be unstable and could cause crashes or data loss.
In my mind, the beta program provides an early window into how the market will receive the product release. It generally happens at the very end of the development cycle. I generally run small-scale beta programs as follows:
- Recruit 10-20 beta testers to match the primary and secondary personas that the product is designed for
- Do a kickoff meeting (either one on one or in a group) to set expectations on what’s in the new release, and how we expect to collect feedback from beta testers
- Ask beta testers to use the product in the target environment of use
- Schedule a call with each tester on the phone 1 week into the program, to ensure everything is going smoothly
- Use phone calls and email to keep track of progress during the program
- At the end of the beta program, schedule a phone conference or an in person debrief to collect feedback.
Since beta programs occur at the very end of the development cycle, typically weeks before the target release date, it is really only useful for testing things that can be iterated right before the release: positioning and messaging, delivery and support mechanisms and the like. There is a great recent post on beta programs by Dave Daniels of Pragmatic Marketing that outlines all this – do take a look, it brings into sharp relief many of the questionable practices a lot of software companies take for granted.
Findings from beta programs can also be used to pull a release if (gasp!) a customer discovers a fatal bug that the QA department failed to find. Lastly, it can also be used as a vehicle to collect customer feedback for the next release. It is NOT a vehicle for usability testing – it is way too late in the game for that! Usability studies (whether in lab or extended use tests) should be done early in the development cycle, before the product is finalized and when there is still time to effect change.
Using beta programs to test positioning is a great idea. One can save a lot of money in marketing programs by iterating the messaging with target buyers until winning messages are arrived at.