War Stories - How I turned around a paralyzed Product organization into a rapid delivery engine
How telling a product management organization "value doesn't matter" was the best possible thing I could've done for them.
A while ago I was given the responsibility of improving the Product Management organization at a startup. It was a Series B company with about 200 people, of which ~50 people were in the development organization. The group of ~10 Product Managers had their leadership recently depart, and they needed someone to step in to help guide them through that transition.
I knew some of the challenges and opportunity areas from the engineering perspective first-hand and through constant feedback from the engineering team.
There was never enough “ready” work to do. The Product Management organization had a difficult time defining and prioritizing work. As a result, engineers went underutilized for weeks at a time, working on tiny things they found themselves here and there with little value.
The requirements that made it to engineering were ill-defined. The work hadn’t been done to define the boundaries of the problem space or even what the end-state was supposed to be, leaving engineers to fill in massive gaps in requirements.
The “Why?” wasn’t defined or answered, let alone asked. The value, the problem it was solving, and how that item mapped to a larger vision or picture wasn’t defined for the development teams.
Cross-feature interaction analysis and product capability management was non-existent. Work items were done in silos without consideration to how they interacted with existing features, which led to duplicative functionality, or different functionality for the same capability. This lead to large numbers of gaps and quirks in the product overall that reduced quality and led to difficult defects - not due to implementation errors, but due to thought not being put into the intended change itself.
Explicit functional boundaries prevented gap filling. The organization was afraid to step on toes or interfere with another function’s work, so boundaries formed, leading to decreased collaboration. Of course, because the work that did get handed off had issues, it meant an ever-increasing burden on Engineering to fill in all the gaps as the last function in the value stream capable of fixing issues.
These challenges were at best slowing down delivery of everything, and at worst completely stalling efforts to deliver value.
Identifying the issues
When I came on board to the Product organization, I had my own hypothesis and opinions of what the issues were, but I did my due diligence:
I read through chat histories, meeting notes, and other documents to understand how decisions were being made
I had 1:1s with every product manager and adjacent functions like Marketing, Privacy, Customer Success, and Solutions that interacted with Product to better understand the issues and challenges from their perspectives.
I mapped out the relationships of all of the issues and their causes, influencers, and effects to better understand the first, second, and third effects.
I conducted surveys and other analysis of sentiment.
From that, I saw some clear root issues that were creating a complex web of spiraling - a vicious cycle of negative feedback that reinforced itself the longer it went on.
The Product Managers didn’t have a true understanding of the product or user.
Without understanding how the product truly worked or what the user truly needed, the Product Managers were afraid to make decisions because they couldn’t figure out the effect of it. When they did make a decision, it often broke some user workflow or had some issue they didn’t consider because of their lack of comprehensive, true understanding of the user, their journey, and their work. It was like the blind leading the blind.
There wasn’t any formalized mechanisms for the organization to get and leverage feedback. Certain members occasionally did interviews, or looked at a competitor, or read an article, but very little was baked into any formal feedback loop. Whatever knowledge we did obtain was silo’d and lost in random slack messages instead of becoming a concrete learning that could be used in the future to make decisions.
The Product Managers wanted to be 100% correct, all the time.
Being wrong was viewed as a terrible thing. In fact, most discussions revolved around how to not be wrong, which naturally led to doing further analysis, measurement, and thinking to avoid being wrong. The quantification of the negative impact of being wrong wasn’t even being considered. Being wrong was avoided 100%, even for the smallest negative impact. Some of this was due to individual Product Manager personalities, and others were due to fear of punitive measures if something went wrong.
The Product Managers over-indexed on working on the highest value item.
Not just value, but specifically the highest value. This meant a lot of discussion, argument, and analysis to intimately understand the exact value of all items being considered to eventually be able to decide which item specifically had the most value. This, of course, increased decision-delay.
The Product Managers wanted everyone to agree before they decided anything.
The Product Management culture was one of consensus - full alignment and agreement was required before anything moved forward. Because nobody wanted to be the one that was wrong, even the smallest disagreement or concern raised derailed a plan to move forward. Few people made decisions, instead opting to have another meeting or touchpoint to get further alignment and clarity. This meeting, of course, was scheduled for a week or two after, which added wait time to even the smallest discussions.
If disagreement persisted, even by one person and even if it was an otherwise minor addressable element, it meant that the item was disregarded. While this was fine if it got replaced by another valuable item, the issue was it got replaced with nothing, which further exasperated the lack of delivery of anything.
The Product Managers cared too much about optics.
There was a massive pressure to concretely measure the specific impact on a particular metric with exacting precision. The go-to method was the A/B test. Product Managers were pushing for every single change to be A/B tested. Despite this, features lacked basic instrumentation because Product Managers generally handed off those responsibilities to the Data Science team.
This of course, meant not only the setup of the A/B test adding time to development efforts, but also monitoring and running and interpreting results. Because the Product Managers as a group weren’t particularly organized or familiar with the data side, the A/B test interpretation fell almost entirely onto the data science function. Because a proper A/B test requires a stable environment, this also meant follow-up iterations were effectively blocked until the A/B test was completed.
In short - analysis paralysis ruled the day. All of these specific issues fed into each other like a vicious cycle. Because of a lack of understanding, product managers made mistakes. This made them more scared to make future decisions, so they tried to analyze more and measure more to prove they did a good job. Because they tried to analyze more before they made any decision, they slowed down their decision making. This increased pressure on the product managers to make sure whatever decision they did make had high value, which further led to more analysis and delays. This decreased confidence so they tried to get all of their stakeholders and peers to agree on every aspect of a change before moving forward, which meant valuable items with minor concerns got disregarded, further leading to slower decisions.
I knew I needed to break this cycle.
The Gordion Knot of Analysis Paralysis
The Gordion Knot is a legend about a complicated knot.
The belief was that whoever untied that knot would go on to rule all of Asia. Alexander the Great came along to untie it. Instead of painstakingly and slowly untying the knot as expected, he just cut it in into pieces with his sword.
It’s a metaphor for intractable problems and the solution spaces that are possible if you don’t make yourself beholden to perceived constraints.
I thought hard about the situation and the culture. I spent several weeks diving deep into the problem space that the organization found itself in.
Fear was the root cause of a lot of these issues. Fear of being wrong. Fear of looking bad. Fear of having others disagree. Fear of being uncertain. Fear of making mistakes. Fear of punishment. The only way to remove the fear is to remove the disincentives.
I created a bold and, frankly, insane, 120-day plan. One that I kicked off at the next Product Management all-hands.
Setting the tone
That Monday, I said clearly and in no uncertain terms to every Product Manager in the company:
I don’t care about the value. The value doesn’t matter. Stop considering it. Stop thinking about it.
I could see the wheels turning in all the product manager’s heads. Their whole job is oriented around delivering value. If value doesn’t matter, what does that mean for them and their careers?
I can see the wheels turning in your heads, too. I can hear the cries of “blasphemy” from all of you readers who are product managers that have been trained on the belief that the highest value is the only thing that matters.
But the fact is, it’s not.
The whole point of a Product Management is to deliver value. The key first word in that is to be able to deliver. It’s not enough to talk about value, or compare one value to another, or know what’s valuable in your head. All of the purpose of the role is predicated on the first word - that value is being delivered.
The team’s problem was that in their pursuit of the value, they forgot how to deliver. The muscle of intentionally and sufficiently defining a change, crafting a narrative, influencing others, working with a development team, and putting it out into the world, and then seeing what the effect was had atrophied. Habits of fear and talking replaced it. Under that fog of fear, they went through the motions of product management without fulfilling its spirit. They got slower because they didn’t know how to act without full knowledge. They got slower because getting full knowledge required delivering, which they didn’t know how to do.
It was lik trying to turn a car into tho right direction without moving either forward or backwards. We couldn’t even get out of the parking lot to know whether we were going in the right direction.
We needed to build up the delivery muscle, and to do that, we needed to go through the reps of delivering something - regardless of the value.
Breaking the desire for full confidence
Some of the Product Managers still weren’t sure. They were still fearful of changes they made. It might be fine to ignore value, but what if it has a negative impact?
My response was simple : who cares? If it has a negative impact, we can see it on the basic instrumentation and roll it back. Changes were reversible. Software is “soft”ware for a reason.
Some weren’t convinced.
“Well, that’s not precise enough”. They wanted to A/B test everything to be absolutely sure it wouldn’t harm a company outcome.
I told them that with change comes risk, and with risk comes the possibility of failure. Trying to measure every single step you take to see if it’s taking you in the right direction isn’t due diligence, it’s fear. There’s no place for fearful product management in a startup.
The fact is, at our company, A/B testing had become a crutch to hide a lack of decision-making.
So, I decided to remove that crutch.
I banned A/B tests.
The right tool for the wrong job
That’s not to say A/B tests aren’t useful. They are, in the right context. The situation our company in was absolutely the wrong context. The desire of the product managers to test everything was getting in the way of making progress.
There were proposals to A/B test even the smallest of changes - changing the text of a navigation bar link, adding a new filter to a dashboard list, or adding instructional text to a search bar. There was even a proposal to A/B test the error message of a donation!
Keep reading with a 7-day free trial
Subscribe to Joseph Gefroh to keep reading this post and get 7 days of free access to the full post archives.