The Two Core Competencies

Good product teams must be good at product discovery, which means they must get good at learning quickly.  They need to be able to zero in on the appropriate target customer, identify the key problems to solve for those customers, and typically the most difficult part of all, apply technology and user experience design to come up with good solutions that will solve those problems.

Our general principle is that we want to learn – test our theories – as fast and cheap as possible.  We have two general approaches to doing so: one qualitative and the other quantitative.

User prototyping and user testing:

• User prototypes are very quickly created simulations of the product you’re proposing to build.
• The main reason we create them is to be able to very quickly put them in front of real target users and customers, generally face-to-face, and test the response.
• The big benefits of user prototypes are that they’re quick and easy to create, generally hours or days; they don’t require developer resources to create; and we can get significant qualitative insights into how the product would be used, whether it will address the customer’s problem or need, and whether people would actually choose to use it or not.
• A user prototype is not about statistically significant results; it’s about big insights and rapid learning.

For more on user testing, see https://svpg.com/the-most-important-thing/.

Live-data prototyping and split testing:

• A live-data prototype is real code that’s typically just deployed to a subset of users in some form of an A/B test.
• The big advantage of a live-data prototype is that we can gather statistically significant results and prove something actually works or doesn’t work.
• Another advantage of a live-data prototype is that we can also test them face to face in a user test just as we do with a user prototype.
• The big disadvantage of live-data prototypes is that since it’s real code, it needs to be written by developers and it typically takes days or weeks to create rather than hours.

For more on live-data prototyping and split testing, see https://svpg.com/product-discovery-with-live-data-prototypes/.

Because of our general principle that we want to learn as fast and cheap as possible, this generally means a user prototype, however, for many things we simply can’t learn whether something works without live-data, and in those cases we need live-data prototypes.

When trying to decide which technique is most appropriate for the situation, one general rule of thumb is that we can best prove something works with live-data prototypes and split testing, but we can best understand why something doesn’t work, and most importantly, what it would take to make it work, with face-to-face user testing.

But the bottom line is that as a product organization, we need to get good at both forms of learning.