Build To Learn FAQ

In our last article, we argued that especially in the age of AI, we are all builders, but that there is a very meaningful difference between building to learn (known as product discovery) versus building to earn (known as product delivery).  

Moreover, we argued that as the cost of product delivery continues to drop, the bottleneck and our competitive advantage moves to product discovery.

The article seemed to resonate with many people, and it inspired many conversations as teams realized many were confusing and conflating these two critical activities.

The result was that many people reached out with questions, or engaged in online discussions.

As is our practice when we receive many follow-up questions to an important topic, in this article we’d like to share the most common questions we’ve received, and our responses:

How do we frame our build-to-learn work?

We are starting with a problem to solve, and an outcome to achieve.

The problem to solve could be a customer problem, or it could be a problem that our own company faces, or both.

Success is measured by achieving the desired outcome.

So, we’ll need to make sure we understand the problem, and then discover a solution that will generate the necessary outcome.

In the product model, what is the hard part: picking the problem, understanding the problem, or solving the problem?

Your product strategy is what picks the problem, and this is usually done by your product leaders.  Product strategy is hard in its own right, but when we’re building to learn, all we really need to know is that this is a real problem that needs to be solved.

Obviously we need to understand the problem in order to solve it, but in truth, most of the time, this is not difficult and doesn’t take us much time.  And if we do have misunderstandings about the problem, or about the people that need this problem solved, we will see this very quickly once we start testing prototype solutions.

The hardest part, by far, is almost always solving the problem.  This is especially true when we’re building a commercial product that must not only solve the key problem, but solve it in a way that is demonstrably better than the alternatives (competitors or existing solutions).  

So building to learn is primarily about solving the problem (aka solution discovery), and that’s where we will need to spend most of our time.

In the product model, isn’t product discovery about confirming that the problem is real?

No, and if you’re perceived as thinking that’s your job, you will almost certainly lose the trust and confidence of your leaders.

More generally, it is quite rare for your product leaders or business stakeholders to prioritize a problem to solve that’s not really a problem after all.  Normally these problems are very well known, and well understood by your leaders.  It’s not rocket science to know something is a real problem.  However, it can be very hard to solve that problem.

But how do we know this is the most important problem for us to be working on?

We don’t, and we can’t.  

All we can know is that this is a real problem that needs to be solved, and that’s not hard to know.  

This is the implicit agreement with empowered product teams:

You are trusting that your product leaders and stakeholders will identify worthy problems to solve, and they are trusting that you can solve these problems in ways that work for the customer and the business.

What exactly are we trying to learn?

We are trying to determine if the solution we have in mind will truly solve the problem, and generate the necessary business outcome.

There can be many different reasons why a solution that sounds good might not deliver the necessary results.  

The most common reason is that while we think a solution is good, the customer is less impressed, and unwilling to switch (value risk).  

Sometimes the solution is just too complicated to use (usability risk).  

Sometimes solutions that sound doable turn out to be much more difficult to build than we initially imagined (feasibility risk).  

And sometimes we have a solution that our customers love, but we learn that the solution would not be compliant, or secure, or legal, or something that we can afford to market and sell, or that can be effectively monetized (viability risk).

In product discovery, we are building prototypes of potential solutions, and then testing these prototypes against these four risks. 

The Role of the Product Manager in Build To Learn

Isn’t my job as PM to explain “the why”?

First, this is normally done by your product leaders in the product strategy.  Second, even if it isn’t, it takes just a few minutes to articulate the problem to solve and the measure of success, and why this is important towards achieving the product vision.  It should be obvious that this is not nearly enough to justify a job, and further, anyone on the product team could do this.

Isn’t my job as PM to be “the decider”?

No, the PM is not “the decider,” and it’s quite dangerous to think of yourself in this light.

Realize that each member of the cross-functional product team including designers and engineers uses their experience and judgement to make dozens or even hundreds of decisions every day. 

Instead, think of a surgical team, with several different skills either in the operating room, or readily available if needed: anesthesia, radiology, oncology, orthopedics, internists, neurologists, etc.  

Rather than having “a decider,” the surgical team defers to the person that is best suited for the particular decision.  If there are impacts on other areas, we collaborate to determine the best solution.

Isn’t my job as PM to be “the protector of the team”?

No, your job is not to insulate the team from everyone that comes with ideas and requests, thinking your job is to protect the team so they can do the important work of discovering and delivering a solution.

It is normal for people (e.g. customers, stakeholders, executives) to come to you with specific ideas to solve the problem you’re working on, and it’s also normal that many or even most of those ideas won’t actually solve the problem in a way that achieves the necessary business results.  

However, your job is not to be the person to say no, but rather be an integral part of the product team that can get to a solution that actually works for the customer and for the business.

Isn’t my job as PM to be “the manager”?

No, the product manager is an individual contributor, just like the product designer and the engineers.  You are a builder, not a manager.  It’s essential to understand that “the product manager is not the boss of anyone,” and it’s very important for the health of the product team that the product manager understands this.

So then what is my job?

As product manager, you are a member of the cross-functional product team, and your specific contribution is that you are responsible for the value and viability of the proposed solutions.  You are shaping the solution to ensure customers will buy, or choose to use, your product (value), and that the solution will meet the needs and constraints of your various business stakeholders (viable).

Just as the engineers bring deep knowledge of the technology, and designers bring deep knowledge of the users, the product manager brings deep knowledge of the customers, the data, the industry and the business to help shape the necessary solutions (aka product sense).

The product manager is building and testing prototypes in order to learn whether the particular solution would generate the necessary outcomes (build to learn).

Can AI accelerate discovery like it has accelerated delivery?

Absolutely, just in different ways.

In product delivery we are working to build a production quality solution (built to earn).  

In product discovery, we need to understand the problem, then quickly prototype solutions, and test those solutions against the product risks.  Generative AI is already helping in each of these activities.

The main difference is that in delivery, generative AI plays very much an automation / code generation role.  In discovery, AI plays more of a prototyping and decision support role.

Moreover, in order to make good decisions in product discovery, the product manager needs to develop what we call strong product sense, and generative AI can be a big help in learning this product sense.

What is the role of the PRD? 

The PRD plays different roles in the project model versus the product model.  

In the product model, once we have discovered an effective solution (build to learn), then we need to communicate our learnings to the engineers so they are clear on what they need to deliver (build to earn).  

The primary way we communicate what needs to be built by the engineers is via the prototype we created in product discovery.  This is referred to as “prototype as spec.”  

That said, there are often additional aspects of the specification that are not easily communicated in a prototype, and these are spelled out in the PRD.  This is usually an enumeration of specific use cases, and non-functional requirements (e.g. what are the scale expectations?).

What is most important is that the PRD not be used instead of product discovery, in which case you are back to the project model.  

In the product model, a good PRD supplements product discovery.  In the project model, the PRD is done in lieu of product discovery.

Countless products have failed because a product manager thinks he or she knows what’s “required” only to find later that they were wrong.

What about learning in delivery?

Product discovery is all about learning fast.  But that doesn’t mean we don’t also learn once the product is live in production.

While product delivery is optimized to earn, it’s also true that once we have our product in production, and we have hopefully expanded access to a much broader set of users and customers, we will start collecting more actual usage data than ever before, and of course we use this data to inform further work.

Moreover, the ultimate test of whether we have solved the customer problem and delivered the necessary outcome can only be established once our product is live, so at a minimum we are eager to learn if our work has had the desired impact.

What about the logic of faster output -> faster outcomes?

If you are using the project model, and you can accelerate the production of the output, then there is the hope that if you can get enough changes launched fast enough, hopefully your outcomes will appear sooner too.  This is referred to as the “ready-fire-aim” approach.

However, one of the product model principles pertaining to product discovery is: Test Ideas Responsibly.  

Strong product companies have learned that it’s very easy for even the most loyal customers to start feeling abused because of constant, erratic change.  They quickly realize they are being used as guinea pigs.  And this is not what most of them signed up for.

There are certain users and customers that opt in to rapid testing of changes, but that is product discovery (build to learn).  If a customer has paid for a product but they come to believe that the product team is just throwing things at them in order to see what sticks, then this can negatively affect revenue, reputation, as well as making the job of your customer success people very difficult.

Product discovery has many techniques, both quantitative and qualitative, designed to enable rapid test-and-learn on select groups of users and customers, enabling us to protect our general users and customers from this experimentation.

Who exactly are we testing the prototype with?

This depends on the product risk.

We are testing value and usability risk with users and customers, as they are the ones that must choose to use or buy.

We are testing feasibility risk with our engineers (engineers on our product team, and also engineers on product teams we depend on).

We are testing viability risk with the relevant stakeholders: sales, marketing, legal, compliance, finance, operations, manufacturing, etc.

Note that normally we do not need to test every risk with every prototype on every constituent.  We test the relevant party based on the level of risk.

How do we know if we made the right choices?

Since the product model is all about delivering business outcomes, the only defensible answer is that if your work generated the necessary business impact, then we know your product team made the right choices.

If not, we use the latest data and learnings to see if we can rapidly improve the results.

Where can I learn more about the techniques of product discovery?

There are two essential books: INSPIRED: How To Create Tech Products Customers Love and Continuous Discovery Habits: Discover Products That Create Customer Value and Business Value.  

See the SVPG, Product Sense, and Product Talk sites for relevant articles, training and workshops.