I regularly meet teams that carry a heavy burden of guilt—they don’t experiment as much as they should. They aspire to build-measure-learn, but they mostly design-build-launch. They wish to run dozens of experiments per month, but barely squeeze-in one per quarter (and even that is typically a late-stage “MVP”.) Both the teams and their managers feel they need to do better. I’m sometimes asked to give “experimentation workshops” or to help teams develop an “experimental mindset”.
On the surface there are objective reasons for the scarcity of experiments. The team may lack infrastructure, knowhow, or user researchers and data analysts. The codebase may be convoluted and the launch process slow. There may be insufficient data, or no direct access to the customers. These are all hard problems that take time and effort to solve, but they are solvable.
But if after many quarters of trying you’re still not experimenting, there may be a deeper issue. It’s a strange thing, but many companies that say they want to experiment, don’t really want to experiment. They want the mechanics, but are unwilling to adopt the underlying philosophy. They desire the benefits, without any of the (perceived) costs. They aspire to be modern and scientific, but cling on to old product development approaches. No wonder no one is super-motivated to do the hard work.
In this article I want to talk about some of the core principles that come with experimentation and what it takes to drive the change.
The Importance of Learning
Experiments are a means to an end, and that end is Learning. Specifically we want to learn long before we launch whether we should pursue the idea as-is, whether we should change it in some way, or whether we should drop it. Embracing expiremenation implicitly means embracing uncertainty—admitting that we don’t know what will work and what won’t.
Many companies don’t truly experiment because they don’t feel they have anything significant to learn. I’ve heard various rationales for this. A common one is that we’re developing “must-have” features that the customers are asking for and that the competitors have—what’s there to learn? In other cases the CEO or CPO occasionally meets with customers—often the most difficult and demanding—and develops a perception of what the market needs. There are other cases, but the result is always the same—we know what we need to develop, now let’s do it. Those leaders that urge their teams to be more experimental, are at the same time asking them to execute on strategies full of untested ideas. This sends a dual message. The teams understand that experimenting is important, but what’s even more important is launching.
Every company, no matter how experienced and successful, has something to learn. The ones that realize this have a clear advantage. In the mid-2000s Google launched Google Docs, an inferior product that had a fraction of the functionality of the market leader, Microsoft Word, and none of the polish. Still, GDocs outdid Microsoft Word in four key areas: ease of use, collaboration, version synchronization, and price. Apparently these were the right four things to focus on—millions of people and thousands of companies were willing to switch to Google Docs despite all its limitations. Productivity suites have been around for decades, and still, as Google Docs demonstrated, there was more to learn.
So the first thing to do is to make learning as much of a priority as building. That means being more curious and humble, asking everyone (managers and employees) to trust data more than opinions, and using decision processes that factor in confidence. It also means changing the goals. People are much more likely to adopt experiments if they are measured on delivering outcomes rather than code.
If you’re unsure where your company stands on learning today, consider this scenario. Two product managers are each given a big idea to run with. The first PM mobilizes the troops, drives alignment, pushes the project forward, and eventually launches with great fanfare. The second PM comes back a few weeks into the project with evidence that the idea is not worth pursuing. Which of these PMs is more likely to be promoted?
Check out my workshops to learn hands-on how to implement this and other product management concepts
You Can Both Build and Learn At High Speed
Here’s the second mindset hurdle companies face with experiments—they’re (subconsciously) afraid that they’ll slow things down. Whenever I teach the concept of combining building with learning, inevitably someone will raise their hand and ask this question: “But what about time to market? Isn’t this approach going to be much slower? ”
The answer is No, it’s actually much faster.
There are two ways to explain this:
- Looking at an individual project—If it’s a short project, say a few weeks long, then yes, probably you’ll get the bits out to production quicker if you just design, build and launch (but look at the next bullet to see why you don’t want to do this.) However in larger projects the dynamics change. The quick iterations of build-measure-learn do away with a lot of the overhead of long projects. As you’re always working on an experiment that is a few weeks away there’s no time for design marathons, procrastination, scope creep, and over-engineering. You often end up launching something smaller than you first imagined, which still delivers the outcome you were after.
- Looking at the broader picture—In reality most ideas are simply not going to deliver the outcomes we hope for. If you optimize for building over learning, you’re going to create an awful lot of waste. The faster you build the more unnecessary features and products you’ll launch and will have to maintain. I tried to demonstrate this point using a simulation. It suggests that combining building with learning can create 5-10 times more value, while at the same time reducing how much you build and launch. Another way to think of this is that you shorten time-to-value, measured as the time from when you set an outcome goal, to the time you achieve it.
Experiments Are Not The Only Way to Learn
Scientists don’t just run experiments, they conduct research. Similarly Amazon, Google and Netflix don’t just carpet-bomb their users with experiments for every conceivable idea—that would make for very inefficient learning. There’s no point in adopting experiments without adopting other forms of research.
Generative research
Generative research (also called Explorative research) helps you understand the space you’re working in, find opportunities, and come up with better ideas. There are many techniques: user/customer interviews, diary studies, field research, surveys, and much more. But you shouldn’t stop with users and customers. There’s the entire market to observe—partners, competitors, other relevant third parties, as well as adjacent markets, technologies and trends. So things like competitive analyses, talking to partners, attending market events, and tracking new technologies, can be considered generative research too. Another way to learn is through building. Hackathons, 20% projects, and other methods to let people work on side projects, can generate important insights and ideas. Apple’s Multi-touch and Google’s Gmail both came about this way.
Sign up to my newsletter to get articles like this, plus eBooks, templates and other resources in your inbox.
Evaluative Research
Evaluative research is what you do when you have an idea in your hand and you want to know if it’s worth pursuing. This is where experiments fit in, but there’s more to evaluative research than just build-measure learn.
Think of your ideas as going through three funnels, where bad ideas are filtered out and good ones proceed:
The first funnel—assessment and fact-finding—evaluates the idea without building anything. We ’re trying to answer questions like: Is this idea going to help achieve our goals? How much does it stand to improve the key metrics? What are the costs and risks? How does this idea stack up against other ideas?
Assessment includes techniques like ICE analysis, assumption mapping, goals alignment, business modeling, Amazon’s Working Backwards, and stakeholder interviews. Fact finding is about looking at data and asking whether it shows support for the idea: customer requests, usage statistics, surveys, user interviews, competitive research and so on. Assessment and fact finding don’t generate strong evidence—they’re not enough to decide to jump all-in (although that’s a common practice), but they’re enough to reject or park many ideas.
For a detailed explanation of the model and the various validation methods see my eBook Testing Product Ideas Handbook.
You Can Start Right Now
This is my last, and perhaps, bigger point. Don’t wait for your company to adopt the right mindset, for the data system in place, and for a user researcher to join the team. Instead, consider embedding research and learning wherever you can. If A/B experiments are out of reach, start with generative research, rigorous assessment and fact finding, and cheap experiments. If you’re going to an industry conference where you’re likely to bump into potential customers, conduct an experiment—have a casual hallway chat with 10-12 of them and ask what’s most bothering them today. See if the problem you’re focusing on is in their top-five. Then tell them about your idea and try to solicit a genuine response. The answers will teach you a lot.
Once the goal becomes learning, you’ll find opportunities all around. As you do more generative and evaluative research, share the findings broadly. This will both build your credibility, and get everyone to see the value in learning. You should start seeing more informed discussions and (hopefully) better decisions. Leverage this to establish learning as a priority and set goals for testing and experiments (with headcount and infrastructure funding). The team is likely to get behind it. Engineers, designers, researchers, all naturally favor evidence-based product development. Be realistic—the change will not happen overnight, but try to make incremental improvements every month. As you start showing concrete results—strong evidence and better outcomes, the managers are likely to fall in line as well and delegate decisions more. You may never do it at the scale of Amazon or Netflix, but eventually you too will find yourself building, measuring, and most important, learning.
This! ☝️👏