Product Discovery vs. Product Optimization

As readers of these articles know, I am a big fan of high-fidelity prototyping and user testing on current or prospective customers.  These techniques form the basis of Product Discovery; it’s the key to discovering the minimum viable product – a product solution that is valuable, usable and feasible (see www.svpg.com/product-discovery/).

However, often I’m asked if these same techniques should be employed for all product changes?  While I love Product Discovery, it’s intended for identifying significant new functionality.  This is common both when we’re starting a new product, but also when we’re considering significant changes to an existing product, such as a usability redesign, or enhancing the product to meet the needs of a new class of user.

But often the work we do is not about discovering new products or capabilities, but rather our task is to optimize the user experience and/or business results of an existing product.  This Product Optimization uses different tools and techniques than Product Discovery, and the scope is generally narrower, but don’t let that fool you as the results can be impressive.

The basic tools of the trade for Product Optimization are Web analytics, and A/B testing support (also known as split testing or multivariate testing).

Good analytics depends on having your site instrumented so that you can in fact know what your users are doing on every page as they make their way through your product.  Good A/B testing support lets you try out different approaches with controlled variations, and then quickly see the results of each.

There are quite a few products to help you with each of these capabilities, but with the free Google Analytics (see www.google.com/analytics) and the recently added Google Optimizer for A/B testing support (see www.google.com/websiteoptimizer), there’s no excuse not to at least use this, if not one of the more advanced solutions out there.

Product Optimization is a continuous cycle of controlled experiments deployed to part or all of your user base, with rapid review of the impact, then selecting the best performing options.  We continue this until we’ve either run out of ideas for improvement, or we believe we have reached the point of diminishing returns.

One of the things I absolutely love about Web sites and cloud-based services in general is that the technology for this sort of instrumentation and near real-time reporting is so powerful and easy.

When I first start working with a company, if they don’t yet have their site instrumented in order to enable the analytics or optimizations, it’s really the top priority to correct this.  I explain that they are “flying blind” without this data, in that they really don’t know what’s going on in their site, and their various ideas for improving things are just theories without the data to support them.

You can go out and watch users interact with your site, and I always find this incredibly valuable in the insights you can gain, but you generally can’t, for example, see the difference between a 5% conversion rate and a 7% conversion rate with this technique.  You need relatively large volumes of actual data to see these differences, so generally we do that with live data and an instrumented site.

Typical examples of Product Optimization include working on the customer acquisition funnel – starting with online marketing programs, then landing pages, then customer registration, activation and action (buying or transacting).

I should also mention that Product Optimization works especially well with Agile development teams, as the rapid iterations and relatively minor changes suit the process well.  It’s frustrating when you just have some minor changes to test out but the next release vehicle isn’t coming along for months, and then you have to wait months more to incorporate your learnings.

A well instrumented site, with a dedicated Scrum team, combined with a product owner that understands Product Optimization can take a mediocre (or long neglected) site and make significant improvements to the business results in a matter of weeks or months.

With the tools that are available today, many teams have become quite good at Product Optimization, and it may be tempting to try to use these same techniques for Product Discovery.  However, just as Product Discovery techniques are generally not well-suited to Product Optimization, Product Optimization techniques are generally not well-suited to Product Discovery.  This is due to several factors:

First, with Product Discovery the iterations are usually daily or every few days, rather than every 2 weeks or monthly.  Second, during Discovery you want to try very different approaches and be very open to Pivots (see https://svpg.com/your-business-plan-is-wrong/), not the type of relatively minor changes that we do in Optimization.  Third, in Discovery you want to be out there watching the eyes of the customers and users, and not so much analyzing the data from wider use.  Finally, the full Scrum team is not only overkill for Product Discovery work, but it is actually inefficient use of the team and a prototype is typically a better match for the purpose.

Note that for those working on products that are not Web-based, you can still get this sort of instrumentation and reporting, but you’ll need to do a bit more work and you may have to do the tooling and reporting yourself (not as hard to do as it may sound).  It’s not unusual for installed software products today to gather usage data and then periodically “call home” to send the data back (aggregated and without personally identifiable information to prevent privacy concerns).

So if you’re responsible for a product, you want to make sure you have the tools and skills needed for both Product Discovery and Product Optimization.  They are very different but complementary techniques and every product leader should be capable in both.