Evidence-Guided: Book Resources

Hi, You’ve found the companion page of my book Evidence-Guided: Creating High-Impact Products in the Face of Uncertainty.  If you haven’t already I recommend reading the book, and using this page as a supplementary resource. 

Note: this page is constantly being updated with more resources, so you may want to check back from time to time. 

Top Downloads

Note: All require email registration. Once you’ve signed up you’ll gain access to all resources (no need to signup again).

Chapter 1: From Opinions to Evidence

Planning Waterfall, Plan-and-Execute

The term “waterfall” often refers to project waterfall, a process that breaks projects into a series of stage-gated phases owned by different disciplines; for example Specification → Design → Development → QA → Launch. Project waterfall was largely replaced by Agile development.  However in the book I talk about the Planning Waterfall, a process of using multi-leveled plans, for example Strategy → Roadmap → Project (of course ideally the strategy should not be a plan, but that’s the case in many companies.)  

A related topic is Plan-and-Execute,  explained in this Article: Escaping the Vicious Cycle of Plan & Execute. Here’s a relevant quote:

“Plan & Execute (P&E) is simply the method by which we start a project by creating a detailed plan and then follow with faithful execution. From the pyramids to jet fighters, P&E is how we’ve been building things and it has served us tremendously well.

20th century business management, starting from Taylor’s Scientific management, embraced Plan & Execute as a core foundation. Managers were given the role of planners and employees the role of implementers. Phase-gate/waterfall, roadmaps, gantt charts, business plans and strategic plans are variations on the theme. Over time detailed plans became a prerequisite for funding and the yardstick for project progress. Success defined as completion of the project as planned, on time and within budget.”

Evidence-Guided Development

I’ve chosen the term Evidence-Guided Development (EGD) rather than Evidence-Based Development because I feel our decisions should be based on our judgment, while  guided by the evidence. To flip this around, it’s not about replacing human judgment with data, as some people may think. Another reason not to use Evidence-Based is that Evidence-Based X (for example Evidence-Based Medicine, Evidence-Based Practice) tends to denote choosing occupational practices based on evidence of their effectiveness. For example a medical doctor may choose a particular treatment because past medical research (conducted by other people) shows that this is the most effective treatment for the illness the patient is experiencing. Evidence-Guided Development is about  us doing our own research and experimentation in order to validate particular ideas or opportunities. As each product, market, and company are somewhat different, we can’t rely much on others’ data, and have to generate our own evidence. 

Guiding ourselves based on evidence isn’t a novel idea. Notable methodologies that encouraged experimentation and data collection include: 

GIST vs. Opportunity Solution Trees (OST) 

You may be familiar with Teresa Torres’ Opportunity Solution Trees (OST) as described in her book Continuous Discovery Habits: Discover Products that Create Customer Value and Business Value . I’m sometimes asked what’s the difference between GIST and OST. 

First, to the best of my knowledge GIST was not inspired by OST, and vice-versa. These methods were developed in parallel independently (a there are yet older, somewhat similar models such Gojko Adzic’s Impact Mapping). 

OST and GIST share some things in common:

  • They’re both evidence-guided frameworks. 
  • They both strongly recommend  the empowered team model led by a trio of product leads 
  • They both use a tree-like structure to lead from goals to action 
  • The Outcomes layer of OST is similar to Goals in GIST, although I’d argue that GIST may go deeper into goal creation and alignment. 
  • The Solutions layer of OST is similar to the Ideas layer of GIST
  • The Experiments layer of OST is similar to the Steps layer of GIST 

Differences

  • OST puts heavy emphasis on qualitative user research, while GIST encourages also other forms of research such as market, technical, and competitive research. 
  • OST emphasizes prioritizing user needs (which it calls Opportunities) while in GIST this layer does not exist. The key reason is that the in my experience good ideas can come  from user research and mapping user needs, but they can also come from a variety of other modes or research or investigation, and so I chose not to introduce this limit. A clear example is a new technology arriving (think Generative ML) – interviewing users about their needs will not help here; we have to research the technology, brainstorm ideas, then test the most promising ideas (with users). 
  • GIST puts focus more on choosing ideas using evaluation/validation loops (chapters 3 and 4), while OST calls for prioritizing opportunities.   

In practice you can use OST and GIST together, which what companies do. They use GIST-like approaches to define goals, OST to map out user needs and brainstorm ideas, then GIST idea evaluation/validation to test and develop the ideas.

Chapter 2: Goals

Objectives and Management By Objectives (MBO)

Outcome goals focus on the destination — where we want to end up — rather than on any (theoretical) path to get there. Already In the early 1950s business management thinker Peter Drucker named these destinations Objectives:

“[Objectives] can be compared to the compass bearing by which a ship navigates. The compass bearing itself is firm, pointing in a straight line toward the desired port. But in actual navigation the ship will veer off its course for many miles to avoid a storm.” — The Practice of Management/Peter Drucker 

As John Doerr explain in Measure What Matters, Drucker proposed a visionary approach to management based on objectives: 

“[Peter Drucker] conceived a new management ideal, results-driven yet humanistic. A corporation, he wrote, should be a community “built on trust and respect for the workers–not just a profit machine.” Further, he urged that subordinates be consulted on company goals. Instead of traditional crisis management, he proposed a balance of long- and short-range planning, informed by data and enriched by regular conversations among colleagues.

Drucker aimed to map out “a principle of management that will give full scope to individual strength and responsibility and at the same time give common direction of vision and effort, establish team work and harmonize the goals of the individual with the common weal.” He discerned a basic truth of human nature: When people help choose a course of action, they are more likely to see it through. In 1954, in his landmark book The Practice of Management, Drucker codified this principle as “management by objectives and self-control.”

Incidentally a similar approach to leadership was developed a century earlier by the Prussian army. It is now called Mission Command and employed by many modern armies. 

Impact, vs. Outcomes Vs. Output 

The terms Impact, Outcomes, and Output as we use them today, where first introduced government departments and NGOs to measure the effect of their initiatives. 

  • Output are the things our activities produce, or put-out
  • Outcomes are the short-term benefits we create. In product companies these are positive changes in customer behavior (for example shorter average onboarding time), in system behavior (for example fewer critical errors) or in business results (e.g. bigger average profit per basket).
  • Impact is the long-term benefit that we wish to gain, usually connected to our core mission and strategic business aims. In the book I mention the North Star Metric and the Top Business Metric and measures of  Outcomes

A good  condensed book on the topic is Outcomes Over. Output by Josh Seiden (although note that Seiden speaks only of human behavior changes as outcomes, which I disagree with). 

Focusing your goals on outcomes, and getting your managers to express their goals as outcomes are topics I cover in my workshops. 

Value to Customer  

Evidence-Guided mentions value-to-customer (or value-to-user) a lot. For example overall value-to-customer is what the North Star Metric approximates. For brevity I didn’t go into details, but I recommend reading this article to understand the concept better. 

What is Value: https://itamargilad.com/10-ways-to-build-more-value-into-your-product/ 

Other useful resources:

Basing Goals Off Models

I mention in the book that “Evidence-guided companies derive outcomes and priorities from models”.  Here are some common models: 

  • Funnels—Funnels trace the state of the user/customer through a series of transitions (aka funnels) which only a subset of users will complete. A famous funnel model is Pirate Metrics defined by Dave McClure. Here’s another funnel model example by Josh Elman. 
  • Flywheels—First introduced by Jim Collins in the book Good To Great. Flywheels (AKA Growth loops) explain growth through one or more virtuous cycles or feedback loops.  Growth loops typically rely on network effects and economies of scale in which every new user creates more value or reduces costs for, which in turn attracts even more users. 
  • Growth Engines—In his book The Lean Startup Eric Ries lists three core mechanisms through which businesses grow, which he names growth engines:  Sticky Growth Engine, Viral Growth Engine, Paid Growth Engine.  Each has its own relevant metrics.  
  • Business Model Canvas (BMC) —Developed by Strategizer, the BMC breaks the success of the business into 9 areas, all of which can be associated with goals. 
  •  User Journeys – A user-experience design tool, user journeys map out the steps and experiences that user encounters during usage. Similar to Funnel models, yet less business-oriented, some companies find user journeys helpful to set metrics, targets and goals. 
  • Growth Equations – described here in Lenny’s newsletter

My Ebook — 15 Essential Product Frameworks — explains many of the models above (and a few others) in more detail (signup to my newsletter required).

Objectives and Key Results (OKR)

My book goes into OKRs in some detail, but of course there’s a lot more to learn.  Here are some supplementary resources:

My eBook OKRs Done Right (signup required) has been used by many companies to broadly educate on the concept of OKRs 

  • We practice OKRs in my product workshops, and I offer also keynotes and condensed workshops on the topics of goals and OKR. 
  • Felipe Castro wrote his own very popular eBook The Beginners Guide to OKR (email signup required)  
  • A great explanation of the core concept is provided by the man who invented OKRS in this archive video from the late 70s/early 80s: Andy Grove explaining OKRs at Intel 
  • Radical Focus by Christina Wodtke, is a short and focused book about OKRs, packed with helpful advice. 

North Star Metric and Other top-level Metrics

The goal of defining a very small set of top-level metrics that are the most important, is to create focus and alignment across the company. For a while it was fashionable to say that you should pick just one top-level metric. However in my experience that would almost always lead to picking a business metric such as revenue or profit. To reflect the dual nature of the org’s mission—delivering value and capturing value— I therefore recommend starting by picking two:  

  •  North Star Metric (NSM) – measures overall value delivered 
  • Top Business Metric (TBM) – measures overall value captured 

A related topic is the One Metric That Matters (OMTM). In this article I define three types of top level metrics, and their relationships. Here’s my explanation of the OMTM: 

“Most people first heard of the One Metric That Matters (OMTM) through Alistair Croll and Benjamin Yoskovitz excellent book “Lean Analytics — Use Data to Build a Better Startup Faster”. Croll and Yoskovitz define the OMTM as the “one number that you’re completely focused on above anything else at your current stage”, and that “answers the most important question you have”. Some people (me included) assumed this to be the same as the north star metric, and started using the terms interchangeably, but actually often that’s not the case. The OMTM isn’t about measuring value delivered or captured, it’s about finding a key metric that enables both to grow.”

In his article choosing your North Star Metric,  Lenny Rachitsky lists the top-level metrics used by 40 growth-stage startups. Although the article talks about North Star Metrics,  many of the  metrics quoted are actually Top Business Metrics, or One-Metric-That-Matters (OMTM).  

Product Analytics maker Amplitude is proposing  a North Star Framework (sign up required). Essentially it’s a model to find the most important metric of the company (what I would call the OMTM) and a metrics tree below it.  

In practice companies typically need to create a short Scorecard of top metrics that may include any or all of the three mentioned above, plus 2-3  health metrics measuring the health of the company, customer satisfaction/well being (but see below why not use NPS), and the health of the product or the service. 

Metrics

A great resource on metrics is this series of articles published by Sequoia:

  • Data-Informed Product Building – this is the master article that lists all available articles, covering goals, metrics, data science, building data measurement in your org and more.  Here are some important metrics: 
  • Defining Product Success: Metrics and Goals — goes into NSMs, goals and their relationships
  • Measuring Product Health – This comprehensive article starts with market sizing, the S-curve model, and then covers a variety of core metric types:
    • Usage: MAU/WAU/DAU
    • Ratios: New users/MAU, Sign-ups/installs
    • Retention: Dn/Mn/Wn 
    • Stickines: for example DAU/MAU
    • Engagement: Time spent/DAU, Session/Day, Time spent/Session… 
    •  as well as important ratios such as 
  • Selecting the Right User Metric explains how to pick the metrics relevant to your type of user/customer. 

Net Promoter Score (NPS)

Some companies are tempted to use NPS as a top-level metric (maybe even an NSM). However there’s debate about the reliability of NPS. For details see my two-part article series:  part 1, part 2

Metrics Trees

This article by Mixpanel offers a layered approach to building your metrics tree. 

In this excellent article Petra Wille and Shaun Russell offer a step-by-step guide to building a KPI tree (essentially the same as metrics tree).  

Chapter 3: Ideas

ICE Scoring

ICE scoring is based on Impact/Ease analysis, which has been a staple of product management for many years. However, in this article I explain why Impact/Eese prioritization is not enough and can lead us down the wrong path. The inclusion of Confidence (add by Growth expert Sean Ellis), makes a big difference in the reliability of scoring and the way we can use it.   

For detailed recommendations on how to use ICE, see my eBook – ICE Prioritization Done Right (email registration required). Bear in mind that this is just one way to use ICE, and not the only one. You may wish to tweak or change it. The eBook talks also about some of the misuses and misconceptions associated with ICE. 

RICE — I cover RICE in the eBook above. Here’s the relevant quote:

  • RICE is a derivative of ICE invented by Intercom (article). It adds a fourth component: Reach—how many users/customers will be impacted. Many ICE practitioners, me included, argue that Reach is simply a component of Impact, and not necessarily a component you always want to factor. For example, an idea that only impacts a small, but very highly-engaged subset of users can be of high impact although it’s low in reach. 
  • Still, RICE is very popular, I believe, because it helps evaluate Impact, the most difficult of the three parts of ICE to estimate (we practice estimating impact in my workshops).
  • If you like RICE you can use it instead of ICE, but I recommend staying critical whether the extra value is creating value and not just extra work. 

Evidence Scoring

In this article I go deeper into the various types of evidence and why they can be considered strong or weak Evidence scores — The Acid Test of Your Ideas.  This article preceded the Confidence Meter, but already help me form the grouping of evidence.  

The Confidence Meter

You can download the Confidence Meter as a spreadsheet-based calculator here 

Chapter 4: Steps

Learning

In this article I explain the concept of Learning Milestones, which underlies steps.  And in this article I explain why learning is key for success and why so many orgs experiment, but don’t learn. 

The AFTER Model 

In Evidence-Guided I present the AFTER model: Assessment, Fact-Finding, Tests, Experiments and Release-results.  In this eBook: Testing Product Ideas Handbook I present the model and each of the validation techniques included in it in more detail (email registration required). The eBook also talks about result analysis. 

For a deeper understanding of the concepts of build-measure-learn and validated learning I recommend reading The Lean Startup by Eric Ries. 

Product Discovery

Product Discovery is a term introduced by Marty Cagan (here’s the original article from 2007). The key point is that product teams have two do two types of distinct activities in order to product products that are valuable, feasible, usable, and viable: discovery and delivery.  Cagan explains the concept in depth in his book Inspired (another must-read), as well as in many articles. 

As Cagan himself explains, Discovery is an umbrella term encompassing a variety of techniques: 

I found the term “discovery” from the pharmaceutical industry for the process they used to come up with a viable medication.  Unlike the software industry, the pharmaceutical industry acknowledged the risk up front.  They expected that many if not most of their medications would not end up being safe and effective enough to manufacture, distribute and sell.

I also liked the term “discovery” because it was technique agnostic.  There are any number of techniques used to help with discovery, and new ones emerge all the time.  An MVP is a discovery technique, as is design thinking, as is customer development, as is “Jobs To Be Done.”

This point is sometimes lost as some folks associate product discovery with one, or a small-set of techniques only (for example user research) . That’s part of the reason I created the AFTER model. 

Switching from discovery to delivery — Core to product discovery is the notion that at some point the team validates the idea enough to switch to delivery. In this article I explain how to use the Confidence Meter as well as the cost and risk of your idea, to determine if you’ve validated enough.   

Chapter 5: Tasks

The Role of Product Management 

Although Evidence-Guided is a book most likely to be read by product managers, I wanted to write a book for product people, product teams, and product orgs, rather than for a specific discipline. 

Still PMs play an important role in processes described in Evidence-Guided, and are central for their success. Although product management has been around for around 40 years (started in Microsoft, HP, and Inituit in the 1980s, and haling back to Brand Managers of the 1960s), it is still a discipline that is often misunderstood or subject to company interpretation. A lot has been written about this topic, here are some resources.   

  • Marty Cagan describes the PM as someone who brings to the team deep knowledge of: Users and Customers, of the Data, of the Business, and of the Industry. 
  • My own article covers what role PMs ought to have (in my opinion), and whom they should report to. 
  • Ken Norton talks about the skills required of PMs. 

The GIST Board 

You can download my GIST Board Template (email registration required) in various formats: Google Slides, Miro, or Powerpoint 

Chapter 6: The Evidence-Guided Company

Strategy

Strategy is outside the scope of the book Evidence-Guided, but in chapter 6 I am showing a model that includes collecting Opportunities and Threats, validating them and discovering strategic ideas, which in turn need further validation, product discovery and eventually delivery, with a think-big-but-start-small mindset. I generally call this approach  multiple strategic tracks (MuST). I’d argue that many strategic moves were discovered this way rather than pre-conceived and executed.  

Roadmaps 

Roadmaps have become somewhat of a contentious topic in many companies, but they’re still widely practiced (even if in the more modern format of Now, Next, Later). In the article: Your Roadmap Isn’t really a Road Map I try to list some of the problems with roadmaps and discuss the alternatives (outcome roadmaps – covered in the book).  

Big Projects

Big projects, and especially “the project that’s too important to fail” are a massively destructive pattern, as I explain in The Big Project Syndrome.  The problem is not with having big initiatives or big bets, it’s with starting big and with skipping idea evaluation and discovery.  

Chapter 7: Scaling GIST

The chapter already has many external references. Here they are again: 

Chapter 8:GIST Patterns

The chapter already has many external references. Here they are again: 

Chapter 9:Adopting GIST 

In the book I list multiple challenges, misconceptions, and objections. A compounding problem is that many organizations are used to optimize for output rather than outcomes, turning product orgs and product teams into feature factories. This is true for the development teams themselves. In this article I suggest a continuous-improvement way for moving from output to outcomes. 

Even a deeper root cause is company mindset (often referred to as company culture). As you start your transition to evidence-guided development, it may be worth first assessing the mindset of your company and determining how much of an uphill battle are you facing.  

About the Book

Here’s a brief explanation as to why  I wrote a book about evidence-guided development, which also includes some of the history. 

If you have feedback on the book, please send it to info@itamargilad.com . 

Working With Me and Staying in Touch

Courses—It’s one thing reading about product approaches and techniques, and another practicing them hands-on and critiquing the results. For this reason I offer courses (live and self-paced) to help build up your knowledge and skills. I have workshops designed for product leads, for team members, and for executives. Check out: EvidenceGuided.com/courses  

Keynotes—If you need help inspiring and informing your colleagues and managers, I’d be happy to share some of my experiences and learnings based on over 25 years of product development. See: itamargilad.com/keynotes 

Consulting—If you want to talk and get my advice visit: itamargilad.com/consulting

My Newsletter—In my monthly newsletter High-Impact Product Management I share articles, news, and new tools. Sign up here: itamargilad.com/newsletter

I post regularly also on LinkedIn and X/Twitter