Most teams want to work on something useful for customers. In the same vein, most product leaders also want to offer something useful for customers. That's how products fulfill the organization's strategy.
Yet, I see so many product leaders outsource their product thinking to LLMs.
Why? Because it's hard to discover what customers want. You might have to do product research, or conduct some sort of survey. And it's easy to ask an LLM.
But that ease is precisely the problem. The LLMs have generic and “old” data. Yes, even if you pay for a model, all that training data is a couple of years old. Unless you upload any customer research you've done for your products with your customers.
Unless you guide them carefully, LLMs do not differentiate between what your customers need and want and what your competitors might offer.
As Jim Grey says, this is a matter of judgment. (See his post Judgment is the skill that matters most in the AI era.)
How do we get that judgment? With short feedback loops and practice.
Short Feedback Loops Matter Even More with LLMs
Any LLM will offer you more possible ideas than you might imagine. That's because they stole and/or “hallucinated” those ideas. (When my children “hallucinated,” they lied. Why are we being so nice to the LLMs who stole other people's ideas? I digress.)
But if you plan to use an LLM to generate possible ideas, you will somehow have to test those ideas and see if your customers want any of them.
That's a big problem. Because, unless you have a collaborative team that collaborates with an LLM, your feedback loops are probably too long to be useful. (See How to Use Value Stream Maps to Reinforce Agility & Effectiveness, Part 3 (Collaborative Teams) for more info.)
Longer feedback loops mean the world continues to move without your insight into what your customers want. At some point, your work ages out of usefulness.
What happens then? You might rely more and more on what the LLM suggests, instead of your own thinking based on short experiments and visual progress.
That's why I like short experiments that offer the team practice in several dimensions:
- Creating prototypes and seeing how customers respond. (You don't need all your customers to respond. There are many ways to sample so you get the information you need.)
- Demonstrating interim increments of value
- Start with the problem(s) you want to solve.
Of all these, starting with the problem might be the most valuable. That's why practice helps me so much.
Practice With the Problem(s) to Solve
Here's a short thought experiment. List all the software products you used so far today. (If that's too many, choose just five of them.)
How many of those products delighted you? At all?
If you're like me, maybe one of the five. Worse, several of those products actively irritated me. Why? Because the producers just added “AI” willy-nilly, without considering which problems I want to solve with their products.
That's the judgment part.
I don't want “AI” to interfere with my work. I know how to use the models to do what I want to do—and I do not want them to “think” for me.
Because the models do not think. Instead, they predict the next set of letters or words, all based on data that's a couple of years old.
I have changed in the past two years. (You have also changed!) Two years ago thinking is not sufficient for the problems I have now. Why would I settle for solutions that are two years old when I need current solutions now?
That's why I do not trust an LLM to tell me what my customers (readers) want. I recommend you do not either.
LLMs Can Offer Value
I am using LLMs to help me in my business, because a consulting and/or writer business is a solved problem. I can use the insights from an LLM to choose how to pivot my business so it works better for me.
But if you are in product development, why would you solve a problem someone else already solved? Solving a problem exactly the same way as someone else did offers no value to you or your customers.
Instead, find ways to ask an LLM to mine insights from your unique data. But trusting the LLM to tell you what your customers want?
No. That is a human endeavor that requires judgment. Don't fall for outsourcing your product thinking to an LLM. You can't trust the LLM to offer you anything that might delight your customers and attract them to your product.
Find ways for your customers to help you learn what they want. That's the ultimate in judgment.