The Role of Analytics

One of the most significant changes in how we do product today is our use of analytics.   Any capable product leader today is expected to be comfortable with data, and understand how to leverage analytics to learn and improve quickly.

Note: For the purposes of this article, consider the terms: “analytics,” “data,” “KPI” (Key Performance Indicator), and “metrics” to be synonymous.

I attribute this change to several factors.  First, as the market for our products has expanded dramatically due to access globally and also via connected devices, the sheer volume of data has dramatically increased which gives us interesting and statistically significantly results much faster.   Second, the tools for accessing and learning from this data have improved significantly.  Mostly, however, I see an increased awareness of the role that data can play in helping you learn and adapt quickly.

In this article I wanted to highlight what I see as the five main uses of analytics in strong product teams.

1. Understand User and Customer Behavior

When most people think of analytics they think of web analytics.  That is but one type.  But the idea is to understand how our users and customers (remember there can be many users at a single customer at least in the B2B context) are actually using our products.  We may do this to identify features that are not being used, or to confirm that features are being used as we expect, or simply to gain a better understanding of the difference between what people say and what they actually do.

This type of analytic has been collected and used for this purpose by good product teams for literally 30 years.  A solid decade before the Internet, desktops and servers have been able to “call home” and upload behavior analytics which were then used by the product team to make improvements.  This to me is one of the very few “non-negotiables” in product.  My view is that if you’re going to put a feature in, you need to put in at least the basic usage analytics for that feature, otherwise how will you know if it’s actually working as it needs to? (see www.svpg.com/flying-blind).

2. Measure Product Progress

I have long been a strong advocate of using data to drive product teams.  Rather than provide the team an old-style roadmap listing someone’s best guess as to what features may or may not work, I strongly prefer to provide the product team with a prioritized set of KPI’s, and then the team makes the calls as to what are the best ways to achieve those goals.  It’s part of a larger trend in product to focus on outcome not output (see www.svpg.com/the-product-scorecard).

3. Prove If Product Ideas Work

Today, especially for consumer companies, we can isolate the contribution of new features, or new versions of workflows, or new designs, by running A/B tests and comparing the results.  This lets us prove which of our ideas actually work.  We don’t have to do this with everything, but with things that have high risk or high deployment costs, or require changes in user behavior, this can be a tremendously powerful tool.  Even where the volume of traffic is such that collecting statistically significant results is difficult or time consuming, we can still collect actual data from our live-data prototypes to make much better informed decisions (see https://svpg.com/product-discovery-with-live-data-prototypes/).

4. Inform Product Decisions

In my experience, the worst thing about product in the past was that is was all about opinions.  And usually, the higher up in the organization, the more that opinion counted.  Today, in the spirit of “data beats opinions” we have the option of simply running a test and collecting some data and then using that data to inform our opinions.  The data is not everything, and we are not slaves to the data, but I find countless examples today in the best product teams of decisions informed by test results.  I hear constantly from teams now how often they are surprised by the data and how minds are changed.

5. Inspire Product Work

While I am personally hooked on each of the above roles of analytics, I have to say that my personal favorite is this last point.  The data we aggregate (from all sources) can be a gold mine.  You will need to dig, and there’s no guarantees.  Often it boils down to asking the right questions.  But by exploring the data, we can find some very powerful product opportunities.  Some of the best product work I see going on right now was actually inspired by the data.  Yes, we often get great ideas by observing our customers.  And yes we often get great ideas by applying new technology.  But a form of leveraging technology is product work inspired by the data itself.

Hopefully you can see the power of the analytics for product teams.  However, as powerful as the role of data is, the most important thing to keep in mind about the role of analytics is that the data will just shine a light on what is actually happening, but it won’t explain why.  We need our qualitative techniques to explain the quantitative results.

I’m hoping you will ask yourself if your team is using data for all five of these purposes.  If not, consider how you can expand the role that data and analytics play for your team.

In the next article, I’ll discuss the main flavors of analytics.