Why Is Product So Hard?

Building great products — ones that deliver high value to the customers and to the company — is extremely hard. This is evident from discussions I hold with product people, who regularly report not being able to create impact. It’s also clear from data: A/B experiment statistics, feature usage data, and startup failure rates, all indicate that even 50% success rate is very high.

But why is product success so hard? And why haven’t all the advancements in product management, UX design, data science, user research, and growth fixed the issues by now? 

The answer is that product development faces three big core challenges that are extremely hard to deal with. We rarely talk about them openly, and beyond surface-level attempts, we rarely try to truly solve them. I’ll try to discuss them now. 

Challenge 1: Uncertainty, Complexity, and Constant Change

Building a technology product is a bit like exploring the edge of space. Every new project is a first of a kind (at least for us) and we’re constantly surprised by how things turn out. This is the challenge of uncertainty, which I wrote a lot about in this blog:

Developing tech products is anything but certain. There are many moving parts in the market, technology, product, and org; many assumptions, dependencies, friction points, and potential snafus. Our well-crafted plans are often no more than a house of cards based on opinions, consensus, and sparse data. 

Embracing Uncertainty: A Modern Take On Strategy, Goals, and Roadmaps

We are often blissfully unaware of our inability to predict. Classic business management requires people with “manager” in their title to create a sense of false predictability, which leads to output roadmaps, feature factories and much waste.

Solutions

Dealing with uncertainty is the area where we’ve made the most progress in recent years. Lean Startup, Design Thinking, Product Discovery and many other methods (including my humble contribution captured in my book Evidence-Guided) offer solutions. 

While the methodologies may differ, the core principles stay the same. To embrace uncertainty we should let go of our production-oriented system of plan-and-execute in favor of a model that combines building with learning, allowing evidence to inform our judgement. We have to tackle uncertainty at every level: strategy, goals, roadmapping, and prioritization. We need to make research and discovery equal-rights citizens (alongside delivery and go-to-market) in every project and every goal. This modern way of developing software not only creates more impact, it also does it much faster. 

Challenge 2: Divergent Needs and Misalignment

Think of your product as a spacecraft coasting across the universe. Like all space vessels, it has thrusters pointing in every direction to help propel and steer it, but for some reason in your case they are controlled independently by separate navigation computers. If all goes well the the thrusters fire in sync and you sail smoothly. However, more often than not the computers are not in agreement on the destination, causing your spacecraft to progress in fits and starts with sudden direction changes and zigzags, while wasting much valuable energy. 

Your product is likewise subject to the push and pull of different needs and interest groups inside and outside the company. There are always these three “strong forces” acting on the product: 

• The Market — Customers, users, and partners naturally require certain things from the product. Interestingly different disciplines within the company think they are the true representatives of the market. 
• The Business — To prosper your company wants a product that will help it close deals, capture market share, catch up to competitors, or fend them off.
• The System — the software you’re building (the spacecraft in the analogy) has to be maintained, upgraded, and fitted with new technology. The engineers who build and maintain the ship are a strong opinion group. 

… but you often also have also these forces pulling and pushing on the product:

• The Company — your company needs to reduce costs, comply with laws and regulations, plan and budget headcount and other resources, enforce a strategy, support its flagship products, build and protect the brand… All of which put demands on your product. 
• Shareholders — investors and shareholders may want to steer the product in the direction that raises the short-term value of the company, makes it an attractive acquisition target, or just fit better in their investment portfolio. 
• Employees and Managers — The people influencing the product may want to shape it in a way that feels more meaningful, more ethical, more interesting to work on, more politically-beneficial, or more conducive of a promotion.

We intuitively think that finding a middle ground between all the worldviews creates the best solution. However compromise (which takes a lot of time and energy to achieve) rarely delivers the best decisions. Decision-by-committee can also amplify the problems of friction, politics, infighting, and constant escalations. More often than not the decision is left to the people in power, who often have a bias in one direction or the other.  

Solutions

A good start is to put in place strategic context to align and focus the company: vision, mission, a strategy, and a small set of top metrics. To be clear, a mission is not a vague marketing message, and a strategy is not a multi-year roadmap of “big bets”. The leadership team has to define the identity of the company — what are we about? How do we create value? — and to discover its strategy — a set of choices on what challenges and opportunities to pursue and a policy on how to best pursue them. This entails being concrete, and saying No to a lot of things, which many leadership teams find very difficult.

The strategic context puts in place the guardrails that align the company, and is the basis for the company goals, but that is not enough. To succeed we must strive to break the walls between the silos, reduce or eliminate departmental goals, and make the work truly multi-disciplinary. Common approaches include delegating most product decisions to product trios at different levels, partnering effectively with stakeholders, and transparently using evidence to back prioritization decisions.

Challenge 3: Org Culture

If your product is the spacecraft, and the managers and employees are the crew, then the organizational culture is the operating system that runs the whole thing. Culture includes the structure and processes of the company, as well as its guiding principles, philosophy and core beliefs. Some companies enjoy a “modern” org culture that allows them to be flexible and adaptive and to do more with their resources. Others use the equivalent of Windows95 and are severely limited in how they can operate. New approaches to product development are either rejected or are subverted to conform to the culture. Special methodologies (SAFe, Six-Sigma, Heavy-handed Scrum) are designed to give a sense of modernity while preserving the old world order. 

Your company is somewhere on this gamut, which means the culture still poses limitations. Ask yourself this: even if you figure out the right thing to build, are you allowed to build it? If the answer is no, that’s likely due to culture.

Solutions

Culture change is very hard. Simply adopting new processes without updating the underlying belief system does not work (see OKRs in most companies). Consultants will be happy to sell you “transformation” or ”change management” approaches like ADKAR and SCARF, management coaching, and mass training. But ultimately this is the hardest challenge to tackle because the org often resists any change that threatens power structures and personal beliefs. Org cultures do naturally shift and fragment over time, but the evolution is rarely planned, nor does it always go in the direction that benefits the organization and its people. 

The most successful tech companies care deeply about their culture and take an active approach to shaping it. During my time at Google, everyone put a high premium on being “Googly”, and “culture-fit” was top criterion in hiring. Amazon and Netflix have very explicit culture codes and values which they broadly broadcast on their recruiting pages (because strong culture attracts strong talent). When Steve Jobs was asked by his biographer what product he was most proud of he said “Apple and the team I built there”.

That’s not to say that bigtech culture is necessarily the best or should be copied. The thing to note is that these companies are willing to devote effort and resources to develop and maintain a culture that truly fits them and their needs. I feel more companies should do that.

But how do you develop your culture?

HubSpot co-founder Dharmesh Shah argues that you should think of your culture as you would of a product:

Culture is a product and your people are its customers.
Said differently, your people/employees are the customers of your culture, which you can think of as a product.
So, every company actually has two products: The one they offer their customers, and the other they offer to their people/employees.

Shah, an engineer by training, takes a very product-like approach to culture development: it has to be built based on employee feedback, it has to have clear use cases and metrics, culture “features” have to be prioritized and pruned, “culture debt” has to be limited, culture is never “done”. Hubspot has an explicit culture code (which you can see here) which it regularly reviews with employees through surveys and pollls, and which it continuously updates (for more see this episode of Lenny’s. podcast with Dharmesh Shah ). 

Culture-as-a-product is a powerful metaphor that helps explain how successful tech companies have arrived at a culture that is entirely their own (and that you can’t really copy). However there’s a chicken-and-egg problem in applying this thinking to your own org — it won’t really work if the culture isn’t ready for it. 

To Boldly Go

If you’re feeling like you’re exploring space by following a made-up roadmap, piloting a spacecraft that has faulty steering and an antiquated operating system, you’re not alone. These challenges are very common indeed.

My goal in this article is not to motivate you to find a better company (this is sometimes the solution, but no org is perfect), nor to make you feel hopeless. Most tech companies I meet want to improve, but are mostly stuck iterating at the surface level. Driving awareness of the root challenges is often an important first step. If it helps, feel free to share this article with your colleagues, and see what emerges in the discussion. You may be surprised to find that many of your co-workers, including managers and stakeholders, share your feelings.