Learning the Truth About “Agile” versus “Waterfall”

When I originally created this post, it drew almost 200 hits in the first day which is an indicator of how important this topic is! In response to that level of interest, I’ve expanded the original video I posted below on YouTube and turned it into a 30 minute Udemy course. The new Udemy course is at the following location:

https://www.udemy.com/learn-the-truth-about-agile-versus-waterfall/#/

This new course on Udemy is free. The original 10 minute video is still available on YouTube at the link shown at the end of this article. Here’s a summary of both the training course and the shorter video…

Many businesses and project managers are faced with a choice of choosing a traditional plan-driven approach (or what is sometimes called “Waterfall”) or a more Agile approach for critical projects which can be a very important decision with significant business impact but there are many misconceptions about “Agile versus Waterfall” that can be very misleading.

One of the big misconceptions is that “Agile” and “Waterfall” are often thought of as binary and mutually-exclusive alternatives. The result of that misconception is that businesses sometimes force-fit their business and projects to one of those extremes when the right solution is to go in the other direction and fit the approach to the project. Sometimes projects require a blend of both approaches – it takes more skill to do that, but it definitely can be done.

Since this is such a critical decision that has such a big impact, I think it’s very important to have a clear understanding of what we mean when we talk about “Agile versus Waterfall”. In this video, I want to try to help you see these two approaches in a fresh new perspective to get the best of both of these two worlds. You can view the video here (it is about 10 minutes long):

http://youtu.be/pp8-jFpa-G4

Learn the Truth About “Agile” versus “Waterfall”

Background

How many times have you heard people compare “Agile versus Waterfall”? It happens a lot, I do it myself, and I keep hearing presentations that talk about how Agile has displaced “Waterfall”. But, if you really think about it, I don’t think that’s a very meaningful comparison and it’s out-of-date – true “Waterfall”, as a methodology, died a long time ago for most projects outside of some specialized areas like construction yet people continue to make that comparison.

The problem is that the word “Waterfall” is used very loosely and indiscriminately. In many cases, when people use the word “Waterfall”, they’re not using it to refer to the specific “Waterfall” methodology that was originally defined by Winston Royce in the 1970’s, they’re using it loosely to refer to a general style of project management that obsessively emphasizes predictability and control over agility.

The word “Waterfall” actually has some fairly specific connotations. For example, a true “Waterfall” project is broken up into phases based on functional activities (design, develop, test, etc.) that typically happen sequentially with phase gates that control the progression from one phase to the next. I think the usage of that methodology for software development today is practically zero, yet people continue to compare Agile to “Waterfall”.

Some Examples of Sloppy Terminology

Don’t get me wrong – I think Agile has huge benefits. I just want people to objectively understand the benefits of Agile versus Waterfall and the sloppy use of terminology to compare the two is often misleading and confusing. Here are a couple of examples I’ve taken from some real world sources to illustrate what I mean by sloppy use of technology when people talk about “Waterfall”:

Blue Line

From the 2011 Chaos Report: “Agile Succeeds three times more often than Waterfall”
The report goes so far as to say, “The agile process is the universal remedy for software development project failure.”

What do they mean by “Waterfall”? (Are they talking about a specific methodology – like the Waterfall that was defined by Winston Royce in the 1970’s or are they talking about a broader range of plan-driven methodologies?)

Blue Line

From an invitation to attend a local Agile group presentation: “It would be so easy if everyone at our companies just used Scrum — or at least Agile. No one would lean on the team for dates and deadlines, and everyone would know that change is a good thing. It’d be one great big happy project management family. But let’s face it — an all-Agile organization isn’t always possible. Maybe you have a Project Management Office (PMO). Maybe you work for a government contractor. Maybe you have regulatory requirements. Maybe you’re the first Scrum/Agile project at your company. Maybe your company simply *likes* it this way. Whatever the reason, Agile teams frequently report into Waterfall organizations.”

What do they mean by a “Waterfall organization”?

Blue Line

When people use the word “Waterfall” like this, I’m tempted to ask, “Which aspect of ‘Waterfall’ are you referring to?”

  • Are you referring to the phase gate approach where a project is broken up into phases and there is a phase gate for approval to transition between phases? I don’t think that approach has been widely practiced for years for software development projects and even Winston Royce himself had reservations about it
  • Are you referring to an over-reliance on documentation? That is a more legitimate comparison because Winston Royce did come out very strongly in support of a lot of documentation, but that shouldn’t imply that an Agile project has no documentation whatsoever.
  • Are you referring to the tendency to plan an entire project upfront before starting the project and then manage changes to the project requirements through change control? That also might be a legitimate comparison, but it also shouldn’t be meant to imply that an Agile project shouldn’t do any planning upfront.
  • Are you referring to the practice of attempting to complete all of the project requirements all at once? Long before Agile became well-known, iterative approaches like the Rational Unified Process (RUP) provided a way to solve that problem and break up a project into iterations.

A More Meaningful Comparison

A more meaningful and more objective comparison is between an “adaptive” approach to project management and a “plan-driven” approach to project management. “Plan-driven project management” is a style of project management that is applied to projects where the requirements and plan for completing the project can be defined to some extent prior to implementing the project. In contrast, an “adaptive” style of project management starts the implementation of a project with a less well-defined plan of how the project will be implemented and the requirements and plan for the project are expected to evolve as the project progresses.

No project is ever totally plan-driven or totally adaptive; you won’t find many projects that start out with an absolutely rigid plan that is not expected to change at all, and you won’t find many projects that have no plan whatsoever of how the project will be done. There is a broad range of alternative approaches between those two extremes as shown in the diagram below:

Agility Range

It is a matter of choosing the right level of upfront planning to be applied to a project based on the level of uncertainty and other factors in the project and it takes some skill to do that.

There is nothing inherently wrong with either of these approaches (adaptive or plan-driven). They both have advantages and disadvantages for a given project and they should be seen more as complementary approaches rather than competitive. Instead, many people see “Agile” and “Waterfall” as binary and mutually-exclusive choices and that causes people to try to force-fit a project to one of those extremes rather than selecting and tailoring the approach to fit the project. For example,

  • If I were to set out to try to find a cure for cancer and I attempted to apply a highly plan-driven approach to that project, the results would probably be very dismal
  • Similarly, if I tried to use an agile approach for building a bridge across a river, the results would be equally dismal

Why does that happen? It takes much more skill to fit a methodology (or combination of methodologies) to a project – you have to know a broader range of methodologies and you have to understand the principles behind the methodologies at a deeper level to know how to tailor the methodology and blend different methodologies together to fit a given situation. Some people are primarily skilled in one particular methodology and tend to implement that methodology mechanically “by the book”. It’s like being a carpenter and the only tool in your tool bag is a hammer.

Overall Summary

The impact of misusing the word “Waterfall” is significant:

  • It causes people to “throw out the baby with the bath water”. By misusing the word “Waterfall” and categorizing all plan-driven approaches as “Waterfall”, people tend to dismiss any form of plan-driven approach and to regard any kind of upfront planning as inconsistent with an Agile project.
  • It has caused a lot of polarization between the traditional project management community and the agile community. The perception is that project managers are associated with the Waterfall approach and, as a result, project management skills are not needed because the Waterfall approach is an out-of-date approach for many projects.

The true Waterfall approach has been obsolete for many projects for a long time (the exception being some selected industries like construction where it is still very useful and relevant). Even prior to Agile, many people were widely using iterative approaches like RUP for software development in place of Waterfall. So, I don’t think comparing Agile to Waterfall is very meaningful any more, but its very difficult to get people to stop thinking in those terms because it has been so prevalent for such a long time.

The difference between a highly adaptive project and a highly plan-driven project is how much of that planning is done upfront in the project rather than being deferred till later. It’s not a black-and-white decision to have a totally plan-driven approach or a totally adaptive approach and it requires some skill and judgment to determine what level of upfront planning makes sense in a given project. When people present this decision as “Agile versus Waterfall” it distorts what the real decision is and makes it look like a binary, all-or-nothing choice and that’s not the case.

Additional Resources

I’ve created a new online training course on Udemy that provides more information on this topic:

https://www.udemy.com/learn-the-truth-about-agile-versus-waterfall/#/

This new course on Udemy is free.

Agile Versus Waterfall

How many times have you heard people compare “Agile versus Waterfall”? It happens a lot, I do it myself, and I keep hearing presentations that talk about how Agile has displaced “Waterfall”. But, if you really think about it, I don’t think that’s a very meaningful comparison and it’s out-of-date – true “Waterfall”, as a methodology, died a long time ago for most projects yet people continue to make that comparison.

The problem is that the word “Waterfall” is used very loosely and indiscriminately. In many cases, when people use the word “Waterfall”, they’re not using it to refer to the specific “Waterfall” methodology, they’re using it loosely to refer to a general style of project management that obsessively emphasizes predictability and control over agility. The word “Waterfall” actually has some fairly specific connotations. For example, a true “Waterfall” project is broken up into phases based on functional activities (design, develop, test, etc.) that typically happen sequentially with phase gates that control the progression from one phase to the next. I think the usage of that methodology for software development today is practically zero, yet people continue to compare Agile to “Waterfall”.

A more meaningful comparison is between an “adaptive” approach to project management and a “plan-driven” approach to project management. “Plan-driven project management” is a style of project management that is applied to projects where the requirements and plan for completing the project can be defined to some extent prior to implementing the project. In contrast, an “adaptive” style of project management starts the implementation of a project with a less well-defined plan of how the project will be implemented and the requirements and plan for the project are expected to evolve as the project progresses.

No project is ever totally plan-driven or totally adaptive; you won’t find many projects that start out with an absolutely rigid plan that is not expected to change at all, and you won’t find many projects that have no plan whatsoever of how the project will be done. There is a broad range of alternative approaches between those two extremes. It is a matter of choosing the right level of upfront planning to be applied to a project based on the level of uncertainty and other factors in the project and it takes some skill to do that. If I was building a bridge across a river, it would probably call for a more plan-driven approach. If I set out to find a cure for cancer, it would probably call for a more adaptive approach.

There is nothing inherently wrong with either of these approaches (adaptive or plan-driven), the problem comes about when people try to force-fit a project to one of these approaches rather than selecting and tailoring the approach to fit the project. For example, if I were to set out to try to find a cure for cancer and I attempted to apply a highly plan-driven approach to that project, the results would probably be very dismal. Why does that happen?

  • People many times see this as a binary choice – the perception is that a project is either Agile or Waterfall and there’s nothing in between. As a result, people feel like they are forced to choose one extreme or the other. They don’t see that there is actually a spectrum of approaches from totally adaptive at one extreme to totally plan-driven at the other extreme. (See my recent blog post on a Hybrid Agile approach for an example of a blended approach):


    https://managedagile.wordpress.com/2013/07/22/a-hybrid-agile-development-framework/

  • It takes much more skill to fit a methodology (or combination of methodologies) to a project – you have to know a broader range of methodologies and you have to understand the principles behind the methodologies at a deeper level to know how to tailor the methodology and blend different methodologies together to fit a given situation. Some people are primarily skilled in one particular methodology and tend to implement that methodology mechanically “by the book”. It’s like being a carpenter and the only tool in your tool bag is a hammer.

The impact of misusing the word “Waterfall” is significant:

  • It causes people to “throw out the baby with the bath water”. By misusing the word “Waterfall” and categorizing all plan-driven approaches as “Waterfall”, people tend to dismiss any form of plan-driven approach and to regard any kind of upfront planning as inconsistent with an Agile project.
  • It has caused a lot of polarization between the traditional project management community and the agile community. The perception is that project managers are associated with the Waterfall approach and, as a result, project management skills are not needed because the Waterfall approach is an out-of-date approach for many projects.

The Waterfall approach has been obsolete for many projects for a long time (the exception being some selected industries like construction where it is still very useful and relevant). Even prior to Agile, many people were widely using iterative approaches like RUP for software development in place of Waterfall. So, I don’t think comparing Agile to Waterfall is very meaningful any more, but its very difficult to get people to stop thinking in those terms because it has been so prevalent for a long time. However, I think it’s useful to understand that when people in the Agile community criticize “Waterfall”, in many cases they are not talking about the specific “Waterfall” methodology, they are really talking about a style of project management that obsessively applies a plan-driven and control-oriented project management approach to projects whether it fits or not.

The problem is really in obsessively force-fitting a project to a given approach (whatever it might be) whether it fits or not. And its just as bad to obsessively force-fit a project to highly adaptive agile approach without any upfront planning if that approach doesn’t fit with the project. For example, I’ve seen an Agile project that has gone for two years without an end in sight before discovering that the scope of the project was much too large for a single Agile team to finish in a reasonable amount of time and the whole project had to be re-planned around multiple Agile teams. Some upfront planning to better define the scope and level of the effort required for the project might have prevented a lot of lost time.

Very few Agile projects are given a “blank check” to do something without any expectations of what the cost and schedule of that effort might be and there is absolutely nothing in the Agile principles and practices that says you shouldn’t do some level of upfront planning for an Agile project. The difference between a highly adaptive project and a highly plan-driven project is how much of that planning is done upfront in the project rather than being deferred till later. It’s not a black-and-white decision to have a totally plan-driven approach or a totally adaptive approach and it requires some skill and judgment to determine what level of upfront planning makes sense in a given project. When people present this decision as “Agile versus Waterfall” it distorts what the real decision is and makes it look like a binary, all-or-nothing choice and that’s not the case.

Managed Agile Development Framework

I’ve seen many people ask a question like “should I use Agile or Waterfall for a project? That presumes that this is a binary, all-or-nothing choice that you have to choose one or the other and not both. It excludes the possibility that there is a hybrid approach that provides the benefits of both approaches.

A few years ago I was responsible for managing a large government project that required meeting some defined cost and schedule milestones but the customer wanted to take an Agile approach to defining the requirements. In response to that project, I developed an approach which I call “The Managed Agile Development” framework that would satisfy those two seemingly inconsistent goals.

  • We were able to successfully build a partnership with the government client in which we did a very professional job of managing overall contractual requirements at the “macro-level”, and
  • Within that “macro level” envelope, we were still able to implement a fairly Agile development approach at the “micro-level”

Managed Agile Development Framework

I’m providing a brief description of how it works here (refer to my book for more details). There are two layers in the framework as shown in the diagram above. The “Macro” layer is plan-driven; but it can be as “thick” or “thin” as you want it to be. The “Micro” layer can be any Agile development approach such as Scrum.

  • The macro-level framework is a plan-driven approach, designed to provide a sufficient level of control and predictability for the overall project. It defines the outer envelope (scope and high-level requirements) that the project operates within
  • Within that outer envelope, the micro-level framework utilizes a more flexible and iterative approach based on an Agile Scrum approach that is designed to be adaptive to user needs

Naturally, there are tradeoffs between the level of agility and flexibility to adapt to change at the “micro-level” and the level of predictability and control at the “macro-level”. It is important that both the client or business sponsor and the development team need to agree on those tradeoffs. The framework provides a mechanism for making those tradeoffs by making the “macro-level” as “thick” or “thin” as you want to fit a given situation.

  • Increasing the level of predictability and control requires beefing up the macro-level and providing more detailed requirements at that level and implementing at least a limited amount of change control
  • To increase the level of agility, you can simply eliminate the macro-level altogether or limit it to only very high-level requirements
  • Other elements of the framework can be easily customized or eliminated depending on the scope and complexity of the project and other factors

A question that often comes up is “How do you handle change control?”. The answer to that question is that you have to design enough slack into the milestones at the “macro” level to allow detailed elaboration of requirements to take place in the “micro” level. However, when there is a significant enough change in the “micro” level that would impact achievement of the requirements in the “macro” level, that should trigger a change to the “macro” level milestones. This general approach can be used on almost any project.

Check out my online training course, “Mastering Agile Project Management” to learn more about developing a hybrid Agile approach and some case studies that show how it has been used successfully.

A Hybrid Agile Development Framework

I’ve seen many people ask a question like “should I use Agile or Waterfall for a project? That presumes that this is a binary, all-or-nothing choice that you have to choose one or the other and not both. It excludes the possibility that there is a hybrid approach that provides the benefits of both approaches.

A few years ago I was responsible for managing a large government project that required meeting some defined cost and schedule milestones but the customer wanted to take an Agile approach to defining the requirements. In response to that project, I developed an approach which I call “The Managed Agile Development” framework that would satisfy those two seemingly inconsistent goals.

  • We were able to successfully build a partnership with the government client in which we did a very professional job of managing overall contractual requirements at the “macro-level”, and
  • Within that “macro level” envelope, we were still able to implement a fairly Agile development approach at the “micro-level”

Managed Agile Development Framework

I’m providing a brief description of how it works here (refer to my book for more details). There are two layers in the framework as shown in the diagram above. The “Macro” layer is plan-driven; but it can be as “thick” or “thin” as you want it to be. The “Micro” layer can be any Agile development approach such as Scrum.

  • The macro-level framework is a plan-driven approach, designed to provide a sufficient level of control and predictability for the overall project. It defines the outer envelope (scope and high-level requirements) that the project operates within
  • Within that outer envelope, the micro-level framework utilizes a more flexible and iterative approach based on an Agile Scrum approach that is designed to be adaptive to user needs

Naturally, there are tradeoffs between the level of agility and flexibility to adapt to change at the “micro-level” and the level of predictability and control at the “macro-level”. It is important that both the client or business sponsor and the development team need to agree on those tradeoffs. The framework provides a mechanism for making those tradeoffs by making the “macro-level” as “thick” or “thin” as you want to fit a given situation.

  • Increasing the level of predictability and control requires beefing up the macro-level and providing more detailed requirements at that level and implementing at least a limited amount of change control
  • To increase the level of agility, you can simply eliminate the macro-level altogether or limit it to only very high-level requirements
  • Other elements of the framework can be easily customized or eliminated depending on the scope and complexity of the project and other factors

A question that often comes up is “How do you handle change control?”. The answer to that question is that you have to design enough slack into the milestones at the “macro” level to allow detailed elaboration of requirements to take place in the “micro” level. However, when there is a significant enough change in the “micro” level that would impact achievement of the requirements in the “macro” level, that should trigger a change to the “macro” level milestones. This general approach can be used on almost any project.

The Agile Project Management Pendulum

The original Agile movement started out as a revolution against the traditional Waterfall methodology which was viewed as very cumbersome, bureaucratic, and inflexible. The need for that revolution was absolutely correct – an Agile approach does offer many advantages where a more adaptive approach is needed; particularly in environments where the requirements are very uncertain and subject to change. However, as in many other revolutions, there’s often a tendency for the pendulum to swing too far in the opposite direction to make a correction.

Pendulum

In particular, a lot of polarization has developed between some people in the Agile community and people in the more traditional project management community.

  • There are many agilists who are entrenched in their perspective that the only way to be “agile” is strictly “by the book” and that there is no need for project management at all – they see project management as a role rather than a set of principles that can be adapted to a broad range of different environments, just as the agile principles can also be applied to a broad range of different environments
  • There are some project managers who are equally entrenched in thinking that traditional, plan-driven, control-oriented approaches are the only way to do project management and have not learned how to integrate an Agile approach into their overall toolkit

The pendulum has begun to swing back towards the middle a bit and there’s less polarization today than there was several years ago, but some of that bias still does exist on both sides of this fence. Some of the progress that has been made over the past few years has been:

  • PMI has recognized the need for integrating an Agile approach with a traditional project management approach and has begun moving in that direction with the PMI-ACP (Agile Certified Practitioner) certification. Although it is a step in the right direction, it doesn’t go far enough, in my opinion. It doesn’t really address the larger question of how a project manager would go about blending together Agile and traditional plan-driven principles and practices in a real-world situation and what role a Project Manager would play in an Agile project to use this knowledge.

    Is the PMI-ACP certification PMI’s answer to the Agile CSM certification? That would imply that the goal of the PMI-ACP exam would be to compete with CSM and train project managers for the Scrum Master role and I don’t believe that makes sense at all. The only way it makes sense, in my opinion, is for a Project Manager to take on a higher-level role in larger projects that require blending together some traditional plan-driven and Agile principles and practices in the right proportions to fit the situation, but that role is somewhat undefined at this point and also not necessarily widely understood and accepted.

  • As Agile begins to be utilized for larger and more complex, enterprise-level projects; there is an increased recognition in the Agile community that an Agile development process like Scrum that works very well at the team level doesn’t necessarily scale very well without some kind of overall management framework and several different frameworks have been developed to fill this need.
    1. The Scaled Agile Framework developed by Dean Leffingwell is an example of a relatively complete approach that incorporates higher levels of project and program management as well as project portfolio management into an overall framework that is fairly Agile from top-to-bottom; however, it is not easy to implement and would typically require a very major transformation for a company to adopt that kind of approach.
    2. For companies who want to integrate an Agile development approach at the team level into a more traditional management framework, the Managed Agile Development approach defined in my latest book and Scott Ambler’s Disciplined Agile Delivery framework are both alternatives that can be used in a more traditional management environment.

It’s time to get past the polarization that has existed in the past and begin to see Agile and plan-driven approaches as complementary to each other, rather than competitive. It’s not an “either-or”, “black-and-white” alternative to adopt an Agile or Waterfall approach as some people have portrayed it; it’s more of a continuous spectrum of alternatives offering different levels of control and adaptivity as needed to fit a given situation. That’s the challenge I’ve tried to take on in my two books on this subject.