Market Research Capabilities and Limitations

If your company is like many, there’s some natural tension between marketing and product. One often controversial topic is the appropriate role in product creation of market research tools and techniques such as focus groups, customer surveys, site analytics, site visits, usability testing/field testing and competitive analysis. Unfortunately I think this is an area of significant confusion, fueled in part by the various camps – those from a marketing background that may see the benefits of these tools, and those from product that see the limitations. The results is that some product teams miss out because they don’t take advantage of the information these tools and techniques can offer, and other teams go astray because they depend on these techniques to answer questions the tools are incapable of.

This is a big topic, but I’d like to try to discuss the major market research tools and consider how they can help you and where they can’t.

Before I get too far, let me say that the tools for market research have improved dramatically in the past decade. Many of the concerns of the past, which I’ll discuss shortly, are addressed by new technologies for easily reaching out to large numbers of users and customers, and for analyzing your user’s activity and behavior – who they are and what they do with your product. That said, there are still some very fundamental, inherent limitations to market research tools, so it’s important to understand that too.

Let’s begin with a summary of the main tools and techniques:

– Customer Surveys. The web has made this so easy and so powerful. Combined with techniques such as conjoint analysis (to help users rank order their preferences) this is so easy and so inexpensive, that it’s a must-do for any product. However, there are two important things to note. First, there is an art to coming up with the survey questions themselves. It is not as easy as it sounds. You should think hard about the questions and context, otherwise you’ll find that people in your company will discount the results because they’ll argue “garbage in, garbage out” which may very well be true if the questions are unclear or biased in their phrasing. Second, you need to set expectations in your company that this data is but one input to the answer – it isn’t the answer. You may very well have every user come back and say “I want X” and it still may make more sense for your company to instead give them “Y”.

– Site Analytics. If your product is a web site, there are terrific tools out there for understanding how your users are using your site. You’ll have to do a little work to make sure your site is instrumented appropriately, but it’s well worth it. Get the site analytics in place early and continually watch and learn – and adjust. If your product is not a web site, you can still usually instrument your product so that it records valuable information about how the product is used and sends that to you. You may have to be clear to your customers that you’re sending aggregated data and nothing personally identifiable, but it’s worth getting that data.

– Data Mining. You’ll collect data from many sources, such as the site analytics I’ve mentioned above, billing and user account information, and your own product’s data. Today there are better tools than ever for analyzing and harvesting that data. Want to know the gender breakdown of people that use some combination of your services? Or the activity level tiers and distribution of a specific user profile? You can usually answer these and thousands of other questions easily and quickly with the new breed of data analysis tools.

– Focus Groups. I have very mixed feelings about focus groups. I basically like anything that puts me in front of users, and they do that, so if handled well there are benefits to be had. But there are some big drawbacks too. First, there is a dynamic that happens when users get together where they influence each other so much that you lose the pure input of each and instead get a skewed representation influenced by the most articulate or vocal attendees. Second, and I’ll discuss this in more detail below, it’s very hard to get useful data about a product unless the customers can actually use the product, and most often these focus groups are conducted prior to the state where that is possible. Third, as with surveys there is an art to conducting these, and finding someone that actually both knows how to conduct these effectively, and yet understands your product domain enough to elicit the depth of conversation you need can be tough.

– Site Visits. Again, I have mixed feelings. There is no real substitute for visiting with your users in their native habitat – home, office, mall – wherever they’re to use your product. It is expensive and time-consuming, yet whenever I do a site visit I realize something I wouldn’t have known any other way. Bottom line for site visits is that depending on your product, you may need to do them, but for cost and time considerations, you’ll want to pick them carefully.

– User Profiling. I love user profiling, especially for product definition and design. Market researchers use profiling too. It’s essential to realize that there is no single “user” and your job is to deeply understand the major types of users out there – those that you have as customers currently and those that you want to have. I’ve written elsewhere in detail about user profiling and how important it is to do, so I’ll just leave it at that.

– Usability Testing. Readers of this newsletter know what a big fan I am of usability testing – early and often. You can also use this tool with existing products to better understand what users really think of your product. Essentially its a way to see their eyes while they use your product – you can gauge enthusiasm or frustration, and watch actions (and not just words). There are tools for doing this remotely, and for recording and analyzing what exactly people do, but this is all just icing on the cake.

– Competitive Analysis. This topic is well covered elsewhere, I just want to emphasize that too frequently product teams write off competitors as clueless, but in my experience every product has at least some things that the product does well, and it’s your job to find these things. Learn from their successes and their mistakes.

With these tools and techniques you can get some very real help answering the following important product questions:

– Do you understand who your users really are? (user profiling, data analysis, surveys, site visits, usability testing)
– How are users using your product? (site analytics, data analysis, usability testing, site visits)
– Can users figure out how to use your product? Where do they stumble? (usability testing, site analytics, data analysis)
– Why do users use your product? (surveys, usability testing, focus groups, site visits)
– What do users like about your product? (surveys, usability testing, focus groups, site visits)
– What do users want added to or changed in your product? (surveys, focus groups, usability testing)

Notice that while these questions are critically important, they do not directly address the fundamental question for most product people: what product to build? This information certainly is an input to the product creation process, but you’re in trouble if you try to steer your product with market research.

The product creation process is about answering these questions:

– What is the product strategy?
– What is the product for, and what does it need to do?
– How should the product be designed?

So how come you can’t just ask your customers what product they want? I’ve said this elsewhere in more detail, but it bears repeating: there are three key reasons why you won’t find customers telling you what to build:

1. customers don’t know what they want – it’s very hard to envision the solution you want without actually seeing it
2. customers don’t know what’s possible – most have no idea about the enabling technologies involved
3. customers don’t know each other – they’re busy enough with their own lives and jobs they don’t have a lot of time for analyzing what needs they have in common with others

As useful as market research tools and techniques are, I know of no winning product that was created by market research. Not Google, not eBay, not the iPod, not MySpace. None. Winning products come from the deep understanding of the user’s needs combined with an equally deep understanding of what’s just now possible. I wish we could just ask customers what they wanted, but if you do you’ll end up with incremental and evolutionary improvements to what they already have (at best) or more likely a random collection of band-aid features, and not the new and dramatically better solution that you’re looking for.

If you’ve already launched your product and if you have a set of active customers, you can learn a great deal from talking to them about what parts they like and what parts they don’t, and getting their views on incremental features. The key is to understand the limitations of each, and that this is all data about refining an existing product rather than conceiving a new one.

So by all means use the market research tools to help you refine your product and make it as good as it can possibly be, just don’t expect the techniques to produce the idea for the next MySpace, Flickr, or YouTube.