Product Discovery: Pitfalls and Anti-Patterns

Article: Product Discovery: Pitfalls and Anti-Patterns

By SVPG Partner Chris Jones

SVPG works with lots of product teams from lots of different companies. Many of these teams believe they are already doing Product Discovery. A few are doing it quite well, but unfortunately most are not. When this happens, it’s often because the team has the wrong understanding of either the intent or the techniques of Discovery.

I’ve noticed several anti-patterns on how Product Discovery is being misused by teams. In this post, I’d like to point out some of the most common ones. If you’re a product leader and recognize your company’s approach in any of these, I hope this post will provide language to characterize the problem and guidance on fixing it.

Anti-Pattern 1: Confirmation-Biased Discovery

Discovery is about finding an effective solution to a problem. Unfortunately, many teams set it up as a mechanism to simply validate their pre-existing ideas. Sometimes this is easy to spot, as when Discovery is used as a rubber-stamp to validate the ideas the team is already committed to. Here, the team pushing a solution through while at the same time appeasing those questioning whether the solution was informed by real customer needs. The team engineers their Discovery to support the product decisions they have already made.

Frequently the problem is more subtle. In this case, teams go through the motions of Discovery, but what is guiding them is their need to more tightly define the various aspects of the product. They move from feature-to-feature using testing to inform what they’ve already decided, but not using it as an opportunity to dismiss or improve ideas that aren’t yet working.

Confirmation-Biased Discovery puts your solution at the center of the process instead of the problem you’re trying to solve. This type of Discovery is not likely to find better alternatives, nor is it a good way to honestly evaluate how the chosen solution is working. If the number and nature of ideas a team begins with closely matches the ideas it ships, the team is likely engaging in Confirmation-Biased Discovery.

Anti-Pattern 2: Product-as-Prototype Discovery

This anti-pattern is really a flavor of Confirmation-Biased Discovery, but arises for a different reason. It happens when teams have a narrow consideration of what constitutes a prototype. For these teams, a prototype is an early version of the product being built, often calling it the “MVP”. It consists of working code that is written from the start to be leveraged in the finished product.  

This type of Discovery is quite slow and comparatively expensive, largely because it requires a high level of involvement from the engineering team. This roots the team in a solution-orientation because at this point everyone is heavily invested in the prototype. By the time they’ve built this prototype, it’s difficult to abandon and expensive to change.

The main sign that a team is using Product-as-Prototype Discovery is that that experiments don’t appear until after there is a prototype with working code. The other sign is a slow testing cadence.

An extreme version of Product-as-Prototype is confusing Discovery with Beta testing. By the Beta phase of a delivery process, most of the decisions have been made and the team is only able to implement small tweaks in the delivered solution.

Anti-Pattern 3: Partial-Team Discovery

Product innovation requires blending of functionality, design, and technology. This means Discovery requires active participation from product management, product design, and engineering (and often others). Many teams drive discovery with only one or two of these roles.

The most common flavor of this is PM-UX Discovery, where engineering doesn’t play an active role in anything except technical feasibility. Teams running this way are doing themselves multiple disservices. First, they’re missing out on what is usually the most common source of good ideas in the company as engineers know what’s possible in ways that other roles often can’t. Second, the handoff from discovery to delivery misses the first-hand learning experience and high fidelity communication of results. These teams must rely more heavily on written documentation. Finally, engineers are often some of the best scientists in the company and frequently have good input on experiment design and flaws.

A less common but more problematic version of this is UX-Lead Discovery. Here, Discovery is seen as the job of Product Design, with little input from either product management or engineering. This has all the problems of the PM-UX Discovery above, but much more. Among other things, Discovery becomes disconnected from the business objectives and go-to-market context of the product. As a result, it either too tactical or too disconnected from the business.  It rarely produces insights that drive the product.

Anti-Pattern 4: One-Dimensional Discovery

Some teams have a favorite, go-to Discovery technique that they use often to the exclusion of all others. These teams approach Discovery through the lens of the tool rather than starting with what they need to learn. This application results in a Discovery process that addresses only a narrow set of product risks.

There are three common flavors of this:

(1) A/B Testing Driven

Quantitative testing is very powerful, especially in cases where a hypothesis requires statistically significant proof (e.g. “if we rollout this new onboarding flow, we will increase user engagement by 15%”). The problem with quantitative testing though is that it doesn’t provide any of the qualitative insights that can explain why something is or isn’t working. Those types of insights are often much more valuable (and actually much easier to obtain), especially early in Discovery.

(2) Usability Testing Driven

Usability testing has been around a long time and there are many tools that make it relatively easy to do. In contrast to A/B testing, it’s a qualitative technique. It answers questions about how well users complete interaction tasks. It should be part of Discovery, but alone is not enough. In particular, usability provides little insight about the value of the solution. It answers questions about whether people can use the solution, but nothing about whether they will use it (or buy it). Minimally, usability testing should be combined with interview questions and conversation to uncover qualitative insights on solution value.

(3) Survey Driven

Survey’s are a seductive tool because they are so easy to execute and because of the quasi-quantitative nature of their results. They can provide insights, but by themselves are not an effective tool for innovation. They are tricky to word in a way that makes them truly open to understanding customer problems. For this reason they often devolve into the realm of asking customers what to build. Better to use them in a manner that supports other quantitative and qualitative techniques.

Anti-Pattern 5: Big-Bang Discovery

This type of Discovery happens when teams spend a long time planning and designing a very small number (often just 1) of experiments. If a team spends weeks preparing and polishing their experiments, it is doing Big-Event Discovery.

Of course the level of Discovery may vary based on the maturity of a product, but generally Discovery is best treated as a continuous process. The best teams run some sort of test, usually qualitative user testing, every week. This keeps them open minded about their ideas because they haven’t had enough time to get attached to them. It also keeps them in continuous contact with their customers and users, allowing them to develop understanding of their real problems.

Anti-Pattern 6: Outsourced Discovery

Many teams look to outside design agencies to do their Discovery. This can be helpful to jumpstart the process, but if teams always rely on outside sources, the customer knowledge that Discovery develops doesn’t accrue into the team. Teams learn best through direct interaction with their customers, not through reading reports from of an outside agency.

A similar problem happens when companies set up internal “Innovation Labs”, responsible for driving innovation and passing their findings into other teams that execute on the ideas. This creates a handoff problem between the insights and implementation. It also sets the expectation that innovation happens only in specific parts of the product organization. The best teams realize that innovation is everyone’s job.

Discovery Done Right

Doing Discovery correctly is not easy, but it’s critical to finding solutions that genuinely work for your customers and for your business. If your team is following one or more of these anti-patterns, it’s time to make some changes.  Keep these important things in mind:

• Discovery requires an open mind. By definition, “discovery” means you don’t know the answer when you start. You must approach it with an openness to kill or dramatically alter our ideas based on you learn
  
• Discovery tackles product risks. These risks go beyond simple usability or feasibility. They also include a product’s value and it’s ability to meet business needs.
  
• Discovery is cross-functional. Good innovation and good communication about the product to be delivered requires active participation by product management, product design, and engineering.
  
• Discovery requires judgement. You must test the things that matter, and test first the hypotheses that will cause us to fail.
  
• Discovery prioritizes fast learning over everything else. The task is not to create reusable code, it is to gain insights into the team as quickly as possible. The most expensive idea is the one that gets built but never used.
  
• Discovery uses many tools.  Different experiments require different techniques. Good discovery makes use of both qualitative and quantitative techniques, based on what needs to be learned.
  
• Discovery is continuous practice. Discovery is best when it’s an ongoing process, rather than a project phase. Good teams are constantly learning by maintaining a weekly cadence of interacting with customers and users.