Our Blog

Practitioner Product Testing: When Bad News is Good News

November 7, 2017 - influencer product testing -
By: Chris Riley

Practitioner product testing helps reduce the risks of your product launch.  Hopefully your product team has done some research into what your target market segment wants, including research into the required features. But you don’t REALLY know how well the product is going to do until it is in the hands of your target market. What they have to say is critical. Paradoxically, the most negative responses you hear might be the most useful.

Fixate offers product testing so vendors can validate their assumptions about their product with their presumptive target market. In order to provide actionable results, we follow a strict methodology for surveying, for the test itself, and the tester interview/feedback process.

Sometimes the product testing highlights minor bugs and missed assumptions. At other times, the outcome is more dramatically unfavorable. Any warm-blooded product manager could be disheartened and even combative when faced with less-than-glowing feedback. There is no way to avoid this. However, overwhelmingly, projects that have given the most value to the product team began with the most negative feedback on the product. How can this be true?

  • Building a product is expensive. If you continue to build a product that users do not respond to, you could be costing the company precious time and money that could be spent in other development projects with a higher impact on your company’s top line.
  • Pissed-off customers are damaging to your brand. If you launch a product that is going to solicit more complaints than value, you will have bigger problems than a product that flops. Your broader brand could be damaged. This is especially true in technical fields where end users very publicly voice their opinions about solutions.
  • Because you might be soooo close. There could be small adjustments to functionality which relate to the buyer’s journey that you would not have been able to identify without exposing the product to your target audience. Sometimes small changes have a huge impact.
  • Your product’s features might have nothing to do with the sales numbers. Often, companies discover that the product’s functionality is not an issue. How the product is launched and how people procure it may have a higher impact on a product’s success or failure. Because so many SaaS products have trials, your prospect’s first impression is the way they experience a trial. A poor user experience with the product trial process alone could be damaging. For example, we recently reviewed a product that required the trial user to not only provide credit card information, but also wait for a sub-one dollar transaction to validate the card even before using the product. This took several days, and the testers found the process so frustrating that if they’d agreed to a trial, they said, they would never start the validation process—or they would just ignore it and drop testing all together. These testers were alienated before they had a chance to look at the product.
  • Your key features are buried. Your product might accurately solve a critical problem. However, the functionality the user needs could be buried so deeply that it takes days of usage or digging in documentation to find it. Therefore, the user can easily assume that the functionality does not exist, and he or she will never discover it. This means that you need to either guide them, or make the features more obvious. But it does not mean your product misses the mark.
  • Maybe you have the wrong target audience. Your assumptions of who can leverage the product could be wrong. A simple switch in target persona will make all the value show up. This will have a huge impact on your go-to-market plan and its success. We once did a product review where the assumed testing persona ended up being off. It turned out their peers in a different department were the ones who would benefit from it the most. They understood and appreciated the product right away.

Chances are you are not entirely wrong about the value proposition of your product. It is providing value and solving problems. But you might be wrong about the impact of how users acquire, test, and use the product. You might also be wrong about some of the details of the target audience and problem/solution fit, and it could have huge and potentially product-killing implications.

If product testing tells you what you want to hear, and agrees with all your assumptions, the value is a green light and a pat on the back. But if product testing tells you what is wrong with your product, the value is saved money, and increased overall effectiveness and success in the open market.

So don’t fret. The impact to your ego could hurt, but it will hurt a lot less than what could happen if the product ends up failing. Practitioner product testing should be a part of every go-to-market strategy, and pre-launch bad news is often the best news for your product’s success.


mm

Chris Riley (@HoardingInfo) is a technologist who has spent 12 years helping organizations transition from traditional development practices to a modern set of culture, processes and tooling. In addition to being a research analyst, he is an O’Reilly author, regular speaker, and subject matter expert in the areas of DevOps strategy and culture. Chris believes the biggest challenges faced in the tech market are not tools, but rather people and planning.