Ten Keys to Product Optimization

Product Optimization is a hot topic.  It can provide real value to teams.  However, many people confuse optimization with product discovery.  I’ve written about this earlier.  Eric Ries of the Lean Startup movement also recently tried to clear up this confusion.

My theory of what’s going on is that there has been an explosion of new tools to help companies with product optimization, especially A/B testing tools and services, and as tool vendors are wont to do, they have been pushing these tools aggressively even for product discovery, and in the process confusing more than a few companies.

I have another article coming soon that discusses how we use live-data prototypes and A/B testing for product discovery.  However, this is not product optimization. We optimize once we’ve discovered product/market fit, but until then, the bigger objective is to discover something that works for the business.  If we do that, then later we can optimize.

All that said, at the right time, product optimization can make a significant difference for a business, and especially for mid-size and larger companies, product optimization typically plays a near constant role.  In this article I’d like to highlight some keys to effective product optimization.

But first, imagine the typical “home page redesign” or “funnel improvement” project.  If you’re like most companies, these are some of the most political projects in the company, as there are dozens of stakeholders and everyone has their opinion.  The relatively minor technical changes are usually overwhelmed by the back and forth between marketing, executives, designers, and product managers.

Weeks or even months pass before something is approved, and very often, the new page launches, and things aren’t really any better.  But there were so many little changes and compromises, it’s hard to know what worked and what didn’t, but that doesn’t stop people from asserting their opinion of what’s wrong, so the cycle continues.  No surprise that these projects are some of the least fun in a company.

Let me instead describe a different process, one with near constant experimentation, where we try changes one at a time, quickly deploying them to a subset of the users and measure the results.  Each experiment shows that the new idea being tested is either better than the current page, worse, or no different.  Sometimes the team gets the results they expected, and sometimes they are surprised, but they are constantly testing and learning and focused on the objective of the page.

Companies that embrace a data-driven culture with rapid test and learn, can move forward steadily, taking the emotion and opinion out of the argument.  As Grace Hopper (one of the original software developers) liked to say:  “One accurate measurement is worth more than a thousand expert opinions.”

So with that vision of a more rational and effective process for optimization (and yes, it really does exist in several companies), here’s my lessons learned for product optimization:

• Define the Experiment.  Agree on the KPI’s up front, and the objective of each experiment, and each test should provide a definitive answer – the new idea is better, the new idea is worse, or the new idea had no impact.
• One Change At A Time.  Generally test one change at a time.  There are ways to actually test multiple changes at a time, but for most companies this is more trouble than it’s worth.  Keep it simple and something the whole company can understand.
• Get Statistically Significant Results As Fast As Possible.   Our goal is to run the experiment and get meaningful data as fast as possible, but we also don’t want to risk our revenue or our brand in case of mistakes.  So we start with only a small percentage of traffic seeing the experiment, and if all goes well,  we ramp up to typically a 50/50 split.  Normally we run experiments for a week.  Online calculators are available for determining how long you have to run the A/B test at a given volume of traffic to get statistically significant results.   See http://www.usereffect.com/split-test-calculator
• Keep Your Eye On The Prize.  A/B testing tools are used in concert with web analytics, and we absolutely need the analytics, but many teams go overboard with the data.  The key here is to measure what’s meaningful.  Lots of time we make minor changes to a page and we get significant results – on that page – but what matters is how many people reach the prize – this might be buying an item, or responding to an invitation, or whatever, but remember that click-thru on a given page only means something if it actually leads to a better result.
• Embrace The Data.  It is essential that you embrace the data and deeply understand what you are measuring and what the caveats are.  Use judgment when deciding what to measure; collect the data that helps you make business decisions.  I encourage product managers and Business Intelligence folks to actually walk through the analytics manually until you have a deep understanding of the data.  Automate the reporting once you have confidence in the data.
• Beware Common Pitfalls.  There are several common pitfalls that can mess with your results of these tests if you don’t pay attention.  First, realize that performance of an operation can play a big role.  Make sure the responsiveness of the test reflects reality.  Another common problem is that many tests that would otherwise require only a day or two to get sufficient data end up needing to run at least a week just so that you can account for weekly trends and traffic spikes. Another common source of grief is automated robots that hit up your site.  One good technique is to run A/A tests continuously.  This should give you confidence in the data.
• Separate New vs. Returning Visitors.  While there are many types of tests each with their own considerations, one that I’d like to call out here is the need to separate results by new and returning visitors.  Their behavior may be different.
• One Source Of Truth.  One thing that doesn’t help is to have all the stakeholders in the company arguing over whether the data is correct or not.  Usually as long as your data is directionally correct, you’ve still got a lot to benefit from.  But make sure that someone in your company is willing to stand behind the data, and that the rest will accept that data.
•  If In Doubt, Take It Out.  Teams often get in trouble because many of the tests don’t actually make much if any visible difference, and since most ideas have at least someone that was arguing for it, teams often push the changes to everyone.  But If a test doesn’t measurably help, don’t launch it.  You will keep the site leaner and clearer.
• Watch For Diminishing Returns.  Beware of the local maximum problem.  At a certain point, you’ll likely hit the point of diminishing rewards.  Getting further usually requires bigger changes, maybe even a round of product discovery.  While rule number 2 above says just make one change at a time, sometimes you’ll want to intentionally break that rule and try something much more dramatic.

Notes:

There are a couple important notes:

• Role of UX.  The role of user experience design is still important but for a product optimization work, the level of effort and the skills required of the UX team are different.  In general, it is much less involved.  Remember that you’re just typically making one design change at a time.  The designer can provide heuristic advice, but it’s not like they’re designing a whole new flow.  Mostly we use our designers for product discovery rather than optimization.
• Role of Marketing.  The collaboration between product and marketing must be close for these types of teams.  If your product optimization team is a dedicated team (see https://svpg.com/dedicated-product-teams/), consider having a dedicated marketing member.

For this sort of product optimization, there are many tools and more recently, hosted services, to facilitate this testing.  There are also several full-service companies that can actually help you run these tests.  Just search for “A/B Testing Tools.”

Whether you develop this capability yourself, or license a commercial tool or service, I hope you’ll give this a try.  It’s really one of the core competencies of a product organization (as well as a modern marketing organization).