AI Product Management 2 Years In
Whenever we talk about Generative AI, we need to be clear whether we’re talking about incorporating this new enabling technology into the products we are building, or whether we’re talking about how this technology changes how we build our products. This article is all about the latter.
At the end of 2023, I published an article containing my predications for the coming year where I spelled out my hopes and fears for each area that I expected to see some real change.
You can read the article and judge for yourself, but as is usually the case, I think overall I was too optimistic about how soon many of these things would happen. The reality is that while there has been a dizzying number of announcements, and countless articles declaring the end of the world as we know it, I have to admit that, thus far, not all that much has really changed. At least as compared to what I was hoping for.
A few months later I co-authored an article arguing that going forward, virtually all product managers will need to be AI product managers, and that contrary to popular opinion, the PM role becomes more essential but also more difficult with generative AI-powered products, not less. The past year has shown this article to be even more important than I had realized at the time.
One thing that continues to muddy the discussion is that we have the three major types of product management: delivery team product ownership, feature team product management, and empowered team product management. Most of what I read is talking about delivery team product owners, or feature team product managers. There is nothing wrong with this as there are many of each in the world; it’s just not where my focus or interest is.
Moreover, it is much easier to discuss the impact of AI on engineers and designers than it is to discuss the impact on product management. Partly that’s because creating code or designing user experiences is much more immediately tangible than creating valuable and viable solutions. But even there, while there’s little question that the tools can speed the process of creating code or designs, that doesn’t necessarily translate into faster and better outcomes.
Generative AI and Product Model Companies
For companies operating under the product model, there are some very important questions that are yet to be answered:
- How does generative AI change the nature of product discovery, and the product manager’s role in product discovery?
- What specific skills become more or less important when product management is assisted with generative AI?
- What aspects of the product management job will generative AI complement, and what will be replaced?
- We talk a lot about the role of creativity for product teams. How will generative AI impact creativity?
- Will generative AI result in more optimizations (minor improvements to existing solutions) or more innovation (major improvements leading to new solutions)?
- Will these changes lead to better outcomes for the product teams?
These are the questions that I’ve been struggling with for much of the past couple of years.
Recently, an academic paper (“Artificial Intelligence, Scientific Discovery, and Product Innovation”) was published by a doctoral student at MIT that is the closest I have found to addressing these fundamental questions.
I consider the paper remarkably relevant, and I’d encourage anyone that’s also interested in these questions to read it, with the caveat that the domain here is scientists working on innovations in materials science.
While inventing new materials is in no way easy, it is much more constrained (by physics) than the type of problems that most of us work on. But the research in this paper is probably the most relevant and thought provoking to the topic of how AI impacts innovation in product teams than anything else I’ve read. While the full paper is long, the first 6 pages summarize the findings.1
There is too much in the findings to discuss everything here, but I will tell you that there’s no simple takeaway. Rather, there is a lot of important nuance in the findings.
As an example, some scientists saw dramatic productivity improvements, and others saw very little. Everything depended on the level of the scientist’s experience. For the most experienced scientists studied, generative AI provided very substantial assistance and value. Yet for the least experienced scientists, the same tools made very little impact.
I’m especially excited by the combination of someone with very strong judgement (product sense) and generative AI tools. But I’m also worried about the prospect of providing those same tools to people that do not have the necessary product foundation.
The most troubling of the findings, to me at least, was the impact of the technology on job satisfaction, especially in terms of their creative contribution, even from the scientists that derived the most value. It’s possible that this is a consequence of the changes and will prove temporary, or it’s possible that this will correct itself by attracting people with different skills and interests to the jobs. I know that I need more time to consider the implications of this.
So, two years in, we’re all still figuring things out. And of course the enabling technology is a moving target as it continues to improve. There’s been real progress, but we’re still just at the beginning of this journey.
But now I would argue we have a better sense of the right questions to ask, and how to go about determining the answers.
Wishing everyone a happy and healthy 2025.
- For those that have access to WSJ content, this article provides a high-level summary. ↩︎
