Monday, May 23, 2016

Agile no longer addresses the fundamental bottleneck

I recently saw a question posed as to whether Agile is dead.

Agile isn’t dead. However, Agile no longer addresses the main bottleneck to delivering value. The bottleneck has shifted. Let’s look at why that is the case and to where the bottleneck has shifted.

First let’s understand what problem Agile was intended to solve. The Agile Manifesto is an artifact of a specific period of time and context. In the 1990s and early 2000s, software delivery problems were rampant (largely due to the heavyweight processes used). Many initiatives were outright failures, and it wasn’t unusual for products to be completed years late or with significant quality problems. Just getting software shipped out the door was the main bottleneck to being able to deliver business value.

The Agile Manifesto and frameworks like XP and Scrum were a response to this environment.  Ideas like working software, iterative development, early integration and frequent delivery were emphasized in order to address this bottleneck of shipping software.

Software developers spent much of the next 15 years improving software delivery by developing and honing techniques like unit testing, test-driven development, behavior-driven development, continuous integration, continuous delivery, continuous deployment, DevOps, etc.

As a result, teams are pretty good at delivering software now. The Agile Manifesto set out with a goal of “uncovering better ways to build software”. In that it has succeeded; software delivery today is greatly improved over the state of the art when the Agile Manifesto was created. In most cases building and shipping software is no longer the bottleneck to delivering business value. The bottleneck has moved, and as a result the problem that Agile addresses is no longer the fundamental system constraint.

The fundamental constraint to delivering value has shifted to building the right thing.  A product is more likely to fail today because no one wants it rather than because of a failure to deliver it or because of quality problems. In other words, most failures today are a failure to reach product/market fit. That is, they are a failure to build the right thing.

Agile doesn’t help to address this new constraint. As Gojko Adzic, Dan North and others have pointed out, Agile isn’t concerned with what to build; it only addresses how to build things.  In other words, Agile is a local optimization. It only addresses a subset of the value creation process: software delivery. Agile principles and practices won’t get you to product/market fit.

Successful organizations in the 21st century will recognize this fundamental shift in the constraint to value delivery and will adapt. The shift is underrecognized. Many organizations are still working on ‘Agile transformations’ without realizing that they are targeting last decade’s problems. What’s required now is a transformation of the product development process into an end-to-end framework enabling building the right thing through a process of product discovery rather than product prescription.

The traditional product development approach is for an organization to target a market, define a product upfront, build it, release it and then see whether it is a success. It’s very difficult to reach a product/market fit with this prescriptive approach. Perhaps it may have worked just enough for organizations with a lot of capital in less competitive environments when business cycles were slower, but today it’s a source of enormous waste and opportunity cost.

How then can we address this new constraint and try to build the right thing? The first step in building the right thing is to avoid building wrong things along the way. This of course requires the ability to recognize something as being the wrong thing.  In other words, it requires feedback loops, learning, and adaptation.  

How can we recognize that something is a wrong thing? A wrong thing doesn’t conform with the assumptions underlying our idea. Therefore we derisk our idea by explicitly enumerating the assumptions that we’re making and then - in order of highest risk - devising inexpensive experiments to try to invalidate each assumption. A failure to invalidate the hypothesis means that further investment in the idea may be warranted.

This transition requires adopting an experimental mindset. Rather than thinking in terms of requirements, we think in terms of hypotheses and in terms of investments in experiments. We iteratively invest a little to validate ideas before making larger investments in ideas. We orient around goals and outcomes rather than around the delivery of features. We use adaptive planning rather than predictive planning.

In essence what is needed is the adoption of a Lean Startup mindset and the creation of a toolbox of techniques for (in)validation. The toolbox includes items such as prototyping, customer & user interviews, market research surveys, usability testing, etc. It’s a process of product discovery achieved by iterating and adapting toward a product/market fit.

Transitioning away from product prescription to product discovery is challenging. Product development transformation is more far-reaching than applying Agile practices and thus is harder to achieve. It requires widespread cultural change, and cultural change is hard.  It requires change at levels and from roles in the organization that have sometimes considered change to be something that applies to others in the organization. For example, changes in governance and in budgeting are required, because an adaptive approach has zero chance of survival in a traditional prepare-a-business-case, pitch it, get approval and budget, measure-deviation-from-plan governance environment.

It’s challenging for an organization and the individuals in it to recognize that this shift in the flow of value has occurred (particularly when preserving the status quo depends on not recognizing the shift). When recognition does occur, it is then challenging to coach and educate the organization in a completely different product development mentality involving significant learning curves for new techniques.

However, therein lies both the challenge and the opportunity. These changes are not simple, but they are also not optional for organizations that want to stay relevant in the 21st century.

So where does this leave Agile? It’s not dead. Agile still has its place, and its practices and principles are still as useful as ever. Agile has done its job and continues to do its job and to evolve. But Agile isn't about building the right thing - it's about building the thing right. Building the right thing is the job of a product development process focused on discovering a product/market fit in a good market.

Update: There's been some good discussion on Reddit


  1. AGILE does not, would not, and should not deal with this case because it's an external pre-requisite.

    You don't want some generic nonsense just to cover what basically seems like stupidity.

    If nobody is going to pay for something, only build it for fun; don't try to make your development philosophy cover economics or business, don't try to make a dictaphone design fine-art or build houses; it's not their purpose...

  2. This comment has been removed by a blog administrator.

  3. I do not understand what the difference between 'value' and 'the right thing' is to you. For me they are the very same.

    1. @darkstar - I think of 'value' and the 'right thing' as fairly synonymous. The 'right thing' needs to deliver business value and to deliver business value it needs to deliver user value (otherwise it's purely extractive).

      I do think that it's helpful to make a distinction for 'value' based on validation. I don't think that it's uncommon for 'value' to be used basically as a prediction. In other words, it is something that someone guesses will be more valuable and therefore it is prioritized higher on a feature list somewhere. I don't consider that to be 'value'. Unless, say, a cost-of-delay can be established for it, it's just a guess. A guess can be useful, but positive validating learning for the item is better than a guess and is closer to the 'right thing'.

  4. This comment has been removed by the author.

  5. I am surprised you don't think Agile is directly related to this. All of these ideas: "working software, iterative development, early integration and frequent delivery" and the sprint retrospective seem to go hand in hand with the Lean Startup ideas of developing a minimum viable product, getting constant feedback and evaluating what to do next. Personally, when I read the Lean Startup I thought the author was saying agile development is so great, it should be used across the whole company and not just software development.

    1. Hi Mike, I do think that they are related, but that doesn't make them identical.

      Lean Startup (as the current most fully developed framework for building the right thing) and Agile go hand-in-hand in a vast majority of cases, and I think that's appropriate. However, I don't think that makes Lean Startup and Agile synonymous.

      Lean Startup brings ideas that are not present in Agile like validated learning, minimum viable product experiments, and the build-measure-learn cycle. Agile is certainly compatible with Lean Startup and strongly associated with it, but I don't consider them to be identical.

      Strictly speaking, I don't think that Agile is necessary to build the right thing either, though it would be hard to imagine something better in most cases. But I can for example imagine a product implemented in a waterfall manner after a validation for the idea had been achieved through, say, a wildly successful Kickstarter campaign. I'm not saying that would be a beneficial approach, but I'm saying that it is possible.

  6. Agile (Scrum) has built in feedback loops at the end of every sprint and at the end of every release to ensure that the team is "building the right thing." If you're not producing demonstrable code and getting it in front of the end users at the end of every sprint, then you're not doing agile (scrum). If you don't have a product owner or an available user base then perhaps a different methodology is warranted.