In this article I want to talk about some of the core principles that come with experimentation and what it takes to drive the change. <\/p>\n\n\n\n
The Importance of Learning<\/strong><\/h2>\n\n\n\nExperiments are a means to an end, and that end is Learning<\/em>. Specifically we want to learn long before we launch whether we should pursue the idea as-is, whether we should change it in some way, or whether we should drop it. Embracing expiremenation implicitly means embracing uncertainty\u2014admitting that we don\u2019t know what will work and what won\u2019t. <\/p>\n\n\n\nMany companies don\u2019t truly experiment because they don\u2019t feel they have anything significant to learn. I\u2019ve heard various rationales for this. A common one is that we\u2019re developing \u201cmust-have\u201d features that the customers are asking for and that the competitors have\u2014what\u2019s there to learn? In other cases the CEO or CPO occasionally meets with customers\u2014often the most difficult and demanding\u2014and develops a perception of what the market needs. There are other cases, but the result is always the same\u2014we know what we need to develop, now let\u2019s do it. Those leaders that urge their teams to be more experimental, are at the same time asking them to execute on strategies full of untested ideas. This sends a dual message. The teams understand that experimenting is important, but what\u2019s even more important is launching.<\/p>\n\n\n\n
Every company, no matter how experienced and successful, has something to learn. The ones that realize this have a clear advantage. In the mid-2000s Google launched Google Docs, an inferior product that had a fraction of the functionality of the market leader, Microsoft Word, and none of the polish. Still, GDocs outdid Microsoft Word in four key areas: ease of use, collaboration, version synchronization, and price. Apparently these were the right four things to focus on\u2014millions of people and thousands of companies were willing to switch to Google Docs despite all its limitations. Productivity suites have been around for decades, and still, as Google Docs demonstrated, there was more to learn. <\/p>\n\n\n\n
So the first thing to do is to make learning as much of a priority as building. That means being more curious and humble, asking everyone (managers and employees) to trust data more than opinions, and using decision processes that factor in confidence<\/em><\/a>. It also means changing the goals<\/a>. People are much more likely to adopt experiments if they are measured on delivering outcomes rather than code. <\/p>\n\n\n\nIf you\u2019re unsure where your company stands on learning today, consider this scenario. Two product managers are each given a big idea to run with. The first PM mobilizes the troops, drives alignment, pushes the project forward, and eventually launches with great fanfare. The second PM comes back a few weeks into the project with evidence that the idea is not worth pursuing. Which of these PMs is more likely to be promoted? <\/p>\n\n\n\n
\n\n\n\nCheck out my workshops<\/a>\u00a0to learn hands-on how to implement this and other product management concepts <\/p>\n\n\n\n
\n\n\n\nYou Can Both Build and Learn At High Speed<\/strong><\/h2>\n\n\n\nHere\u2019s the second mindset hurdle companies face with experiments\u2014they\u2019re (subconsciously) afraid that they\u2019ll slow things down. Whenever I teach the concept of combining building with learning, inevitably someone will raise their hand and ask this question: \u201cBut what about time to market? Isn\u2019t this approach going to be much slower? \u201d <\/p>\n\n\n\n
The answer is No, it\u2019s actually much faster. <\/p>\n\n\n\n
There are two ways to explain this:<\/p>\n\n\n\n
- Looking at an individual project<\/strong>\u2014If it\u2019s a short project, say a few weeks long, then yes, probably you\u2019ll get the bits out to production quicker if you just design, build and launch (but look at the next bullet to see why you don\u2019t want to do this.) However in larger projects the dynamics change. The quick iterations of build-measure-learn do away with a lot of the overhead of long projects. As you\u2019re always working on an experiment that is a few weeks away there\u2019s no time for design marathons, procrastination, scope creep, and over-engineering. You often end up launching something smaller than you first imagined, which still delivers the outcome you were after. <\/li>
- Looking at the broader picture<\/strong>\u2014In reality most ideas are simply not going to deliver<\/a> the outcomes we hope for. If you optimize for building over learning, you\u2019re going to create an awful lot of waste. The faster you build the more unnecessary features and products you\u2019ll launch and will have to maintain. I tried to demonstrate this point using a simulation<\/a>. It suggests that combining building with learning can create 5-10 times more value, while at the same time reducing how much you build and launch. Another way to think of this is that you shorten time-to-value<\/em>, measured as the time from when you set an outcome goal, to the time you achieve it. <\/li><\/ul>\n\n\n\n
Experiments Are Not The Only Way to Learn<\/strong><\/h2>\n\n\n\nScientists don\u2019t just run experiments, they conduct research. Similarly Amazon, Google and Netflix don\u2019t just carpet-bomb their users with experiments for every conceivable idea\u2014that would make for very inefficient learning. There\u2019s no point in adopting experiments without adopting other forms of research. <\/p>\n\n\n\n
Generative research<\/h3>\n\n\n\n
Generative research (also called Explorative research) helps you understand the space you\u2019re working in, find opportunities, and come up with better ideas. There are many techniques: user\/customer interviews, diary studies, field research, surveys, and much more. But you shouldn\u2019t stop with users and customers. There\u2019s the entire market to observe\u2014partners, competitors, other relevant third parties, as well as adjacent markets, technologies and trends. So things like competitive analyses, talking to partners, attending market events, and tracking new technologies, can be considered generative research too. Another way to learn is through building. Hackathons, 20% projects, and other methods to let people work on side projects, can generate important insights and ideas. Apple\u2019s Multi-touch and Google\u2019s Gmail both came about this way.<\/p>\n\n\n\n
\n\n\n\n
Many companies don\u2019t truly experiment because they don\u2019t feel they have anything significant to learn. I\u2019ve heard various rationales for this. A common one is that we\u2019re developing \u201cmust-have\u201d features that the customers are asking for and that the competitors have\u2014what\u2019s there to learn? In other cases the CEO or CPO occasionally meets with customers\u2014often the most difficult and demanding\u2014and develops a perception of what the market needs. There are other cases, but the result is always the same\u2014we know what we need to develop, now let\u2019s do it. Those leaders that urge their teams to be more experimental, are at the same time asking them to execute on strategies full of untested ideas. This sends a dual message. The teams understand that experimenting is important, but what\u2019s even more important is launching.<\/p>\n\n\n\n
Every company, no matter how experienced and successful, has something to learn. The ones that realize this have a clear advantage. In the mid-2000s Google launched Google Docs, an inferior product that had a fraction of the functionality of the market leader, Microsoft Word, and none of the polish. Still, GDocs outdid Microsoft Word in four key areas: ease of use, collaboration, version synchronization, and price. Apparently these were the right four things to focus on\u2014millions of people and thousands of companies were willing to switch to Google Docs despite all its limitations. Productivity suites have been around for decades, and still, as Google Docs demonstrated, there was more to learn. <\/p>\n\n\n\n
So the first thing to do is to make learning as much of a priority as building. That means being more curious and humble, asking everyone (managers and employees) to trust data more than opinions, and using decision processes that factor in confidence<\/em><\/a>. It also means changing the goals<\/a>. People are much more likely to adopt experiments if they are measured on delivering outcomes rather than code. <\/p>\n\n\n\n If you\u2019re unsure where your company stands on learning today, consider this scenario. Two product managers are each given a big idea to run with. The first PM mobilizes the troops, drives alignment, pushes the project forward, and eventually launches with great fanfare. The second PM comes back a few weeks into the project with evidence that the idea is not worth pursuing. Which of these PMs is more likely to be promoted? <\/p>\n\n\n\n Check out my workshops<\/a>\u00a0to learn hands-on how to implement this and other product management concepts <\/p>\n\n\n\n Here\u2019s the second mindset hurdle companies face with experiments\u2014they\u2019re (subconsciously) afraid that they\u2019ll slow things down. Whenever I teach the concept of combining building with learning, inevitably someone will raise their hand and ask this question: \u201cBut what about time to market? Isn\u2019t this approach going to be much slower? \u201d <\/p>\n\n\n\n The answer is No, it\u2019s actually much faster. <\/p>\n\n\n\n There are two ways to explain this:<\/p>\n\n\n\n Scientists don\u2019t just run experiments, they conduct research. Similarly Amazon, Google and Netflix don\u2019t just carpet-bomb their users with experiments for every conceivable idea\u2014that would make for very inefficient learning. There\u2019s no point in adopting experiments without adopting other forms of research. <\/p>\n\n\n\n Generative research (also called Explorative research) helps you understand the space you\u2019re working in, find opportunities, and come up with better ideas. There are many techniques: user\/customer interviews, diary studies, field research, surveys, and much more. But you shouldn\u2019t stop with users and customers. There\u2019s the entire market to observe\u2014partners, competitors, other relevant third parties, as well as adjacent markets, technologies and trends. So things like competitive analyses, talking to partners, attending market events, and tracking new technologies, can be considered generative research too. Another way to learn is through building. Hackathons, 20% projects, and other methods to let people work on side projects, can generate important insights and ideas. Apple\u2019s Multi-touch and Google\u2019s Gmail both came about this way.<\/p>\n\n\n\n
\n\n\n\n
\n\n\n\nYou Can Both Build and Learn At High Speed<\/strong><\/h2>\n\n\n\n
Experiments Are Not The Only Way to Learn<\/strong><\/h2>\n\n\n\n
Generative research<\/h3>\n\n\n\n
\n\n\n\n
![\"\"](\"https:\/\/itamargilad.com\/wp-content\/uploads\/2021\/05\/Screenshot-2021-05-23-at-11.53.08-1024x573.png\")