Discovery – Judgement

To continue with the series on common sources of confusion regarding product discovery, in this article I want to talk about the critical role of judgement in product discovery.

This may in fact be the single most common confusion I encounter.  I think the source of the issue is that so many product people have been led to believe (and wish to believe) that there’s a framework or template for everything.  But product is very much about judgement.

In terms of product discovery, there are four consistent areas where judgement is necessary, and I’ll discuss each of them here:

Assessing Risk

At the core of product discovery is that we have four major risks with everything we pursue: value, usability, feasibility and viability.

However, that doesn’t mean that all four risks are significant or equal.  Everything we pursue has a different risk profile.  And it’s also true that for many of the minor items, none of the risks are significant.  In which case, we just add the work to the backlog and move on.

Is it possible you are wrong, and what looks like a minor risk turns out to be a major risk?  Sure.  And occasionally, you learn that the hard way.  But realize that if you were to treat every risk as a major risk, then a) you’ll move way too slowly; and b) you probably won’t have the time you need for the items that truly are major risks.

More generally, whenever you consider risk, you need to consider consequence.  

In other words, with many risks, if it turns out that you judge something wrong, but the time and effort to correct that mistake is just a few hours of work, then that is a very different situation than when the mistake causes substantial lost revenue, or time and effort to correct.

So it’s essential that when planning your discovery work, you consider each of the four risks, and judge both their severity, and the potential consequence.

Note also that it’s absolutely essential to lean on the other members of your product team when assessing risk.

One of the most common, yet largely avoidable, problems is when a product manager judges technical feasibility risk, without consulting the engineers.  When an effort turns out to take substantially longer than anticipated, this is very often the root cause.

Similarly, it’s important to lean on the product designer when assessing usability risks.  Especially with complex workflows as is often the case in B2B software, it can be easy to underestimate the complexity and associated risks.

Assessing Necessary Evidence

Once we have identified what we judge to be the major risks, then you’ll need to decide how much evidence you need to be comfortable making a decision.

Evidence could span the full spectrum with just qualitative opinions from a small group of users at one end, all the way to statistically significant results in a discovery A/B test at the other end.  In between you’ll find qualitative feedback from a larger group, to quantitative feedback but at the lower confidence level that comes from a smaller group, and many other forms of evidence.

You could always insist on the highest standard of evidence (statistically significant proof) but with most products, you’ll pay a very high price for that in terms of time.

We want to move fast, but we also have a responsibility to protect revenue, brand, customers and colleagues.

So we’re back to judgement.  We need to balance time and cost against the risk and consequence.

For the teams I advise, I like to save the large scale, live-data tests for the truly risky, high consequence issues, and I like to make the bulk of decisions on much faster, albeit lower confidence, qualitative tests.

Assessing Scope of Work

The third dimension to judgement has to do with recognizing the scope of the product work.  This usually maps directly to consequence, but not always, so it’s important to consider this.

Every product team works on a broad range of efforts from simple bug fixes, to performance work, to optimizations, to features, to larger projects to very large initiatives.

Projects and initiatives are, almost by definition, risky.  Just because of the large time and effort that goes into them.  You really don’t want those to fail.

On the other hand, features can range from very minor and low-risk, to very complicated and high-risk.  Even certain bug fixes have the potential to cause real damage.

So, again, you’ll need to judge the situation and determine where you need to spend your time in discovery.

Assessing Necessary Documentation

This last area is a bit different than the others, but it’s a very common issue, especially in a work-from-home world.

One of the side benefits of co-located product teams is that the engineers are so involved in the discovery work that when something is ready to be built, they often know precisely what they need to do.  

One of the side benefits of prototypes is that they do a much better job of describing the necessary behavior than most documents.

Between the prototypes and the co-location, the engineers have already identified sources of ambiguity, and have had their questions answered.

However, when the team is not co-located, such as when some or all of the team is working from home, the engineers may not have been participating in the discovery work as much as before, so when it’s time to build, they may have a lot less clarity.  In this case, they may ask for additional documentation.

The key is that we need to provide the necessary documentation, but we don’t want to provide anything beyond that, because that’s a big waste of time to produce.

The common mistake I find is that companies end up creating a template of this sort of supplementary documentation, and eventually, what begins as very minimal documentation, grows to a big overhead that’s rarely read anyway.

So please don’t do the template thing, and instead, use your judgement on each thing you’re describing.  Talk to the engineers about specifically what they need for this case, and then the product manager and designer can provide this.

This is one of those cases where experience is a real advantage.  An engineering tech lead, or a product designer, or a product manager that has experienced a broad range of product work will have already learned many of the lessons necessary.

When you have a newer team member, one that likely doesn’t have this experience, then we depend on the person’s manager to provide the necessary coaching on developing this judgement.