The Impact of Wishful Thinking in Agile Projects

Have you ever been in an Agile project that was so badly under-staffed that it had no hope of success in any reasonable time? I’ve seen that situation more than once…what seems to happen is that people launch an Agile project without a lot of planning to understand the scope, without doing a schedule estimate to see how long it will take with the current team, and without determining if it is even feasible at all with the current resources. There seems to be a feeling based on a lot of “wishful thinking”, that an Agile team can do anything given enough time.

There are a couple of things that are typically missing in many Agile projects. One is upfront planning…a lot of people may think doing upfront planning to estimate the scope, costs, and schedule and to explore resource tradeoffs is inconsistent with Agile. I don’t believe it is, but you can do as much or as little of it as you want. At one extreme, you can start an Agile project with little or no defined Product Backlog, completely define the requirements as you go along, and have no idea what the overall cost and schedule will be. Or, at the other extreme, you could have a much more well-defined Product Backlog with story point estimates to estimate the costs and schedule, a defined Product Road Map for completing the project, and a reasonably good idea of when it will finish. I am amazed at the number of people who just launch into an Agile development effort without even realizing that is a choice they can make.

There is also typically such a sense of unbridled optimism and wishful thinking that everything will just somehow come out right that permeates Agile projects. Project Managers develop kind of a sixth sense about projects. It’s kind of a sense of smell you develop after you’ve been a PM for a while that you learn to recognize when a project just “smells” bad and is doomed for failure. You develop that sense of smell more rapidly if you’ve been burned a few times in projects that have failed. A good PM learns to recognize those symptoms of failure early.

That sense of impending doom seems to be missing from many Agile projects that are on a path to failure. What seems to happen is that from sprint-to-sprint; the project may be producing good stuff but without an overall plan, no one steps back to see if the project is really going to finish in a reasonable amount of time.

There’s seems to be an opportunity here to integrate some good, old-fashioned, traditional project management thinking with a modern Agile development approach.

Adaptive Leadership

I’ve been a Project Manager for many years and, over the years, I’ve gone through a lot of job interviews, particularly as a consultant where you might change roles every 3-6 months. One of the questions I’ve been asked in interviews is “What is your leadership style?”. My answer to that is I use an “adaptive leadership” approach; that is, I think that’s there’s not just one leadership style that works all the time and you have to adopt a leadership style that is appropriate for the situation.

How does that apply to Agile Project Management? There is a popular stereotype in the Agile community that all project managers are only capable of operating in a “Command and Control” leadership style. I’m sure that is an exaggeration, but it is true that many project managers have a tendency to assume a somewhat directive leadership style. For years, that has been an essential characteristic of many project managers – you can’t just sit on the sidelines and let a project run its course without some kind of direction and leadership. Agile changes that paradigm dramatically by emphasizing self-organization and empowerment of the team and positions the Scrum Master in the role of a “Servant Leader” to support the team rather than leading and directing the team. So, where does that leave the project manager? What value does he/she provide to an Agile team? I think the appropriate answer to those questions is that “it depends”.

A lot of people will say that a project manager can’t possibly play the role of a Scrum Master because the roles are so different. I don’t necessarily agree with that perspective…that perspective is based largely on the stereotype that all project managers are only capable of operating in “command and control” mode. I believe a good project manager has learned over the years to develop an adaptive leadership approach that’s appropriate for the situation and I think that’s very appropriate in an Agile environment.

There is an idealistic Agile view that all Agile teams are totally self-organizing, completely empowered, and require little or no direction or leadership. The team, as a whole, should function on its own without much direction at all – that’s true to some extent, but a more pragmatic view is that all teams aren’t necessarily at that level of maturity and some leadership is needed to help them get to that point. “Adaptive leadership” is an important skill in this kind of environment…a good Project Manager or Scrum Master should be capable of providing a sufficient level of leadership to get the team to a level of self-sufficiency and progressively back out as the team reaches that level.

“Adaptive Leadership” or learning to adapt your leadership style to the situation is a very important characteristic for project managers to be successful in an Agile environment.

Agile “Tunnel Vision”

I am amazed at the amount of “tunnel vision” that exists about Agile. Many people see it only as a development process for single teams and don’t pay much attention to the larger context that the development process operates in. An Agile project always exists inside of some business context and if you look at the project from that business context rather than looking at it only from a development perspective, you might see a very different picture.

A pure Agile approach is best suited for a company that is in the business of producing some kind of software product (an example would be Intuit and TurboTax or Quicken) or where the software plays a very direct role in leveraging the company’s primary business (an example would be Amazon.com). Where that is not the case and the company’s business is only indirectly leveraged by the Agile development process (an example would be an internal IT application development project), it can be a lot more difficult to implement an Agile development process because more adaptation may be required to fit the Agile development process to the company’s business environment.

Here’s the difference – in the product development company, the company typically budgets a fixed amount of development effort to produce and sustain the product and the business decision is primarily about prioritizing how that money is spent on creating new features for the product to get the most business impact. That kind of business decision fits very nicely with an Agile development process approach and it’s relatively easy to implement an Agile development approach in that kind of environment.

In a company that is in some other kind of business where the software development effort plays a more indirect role in leveraging the company’s business, there is typically a very different kind of decision process. An example would be a company whose business is focused on achieving operational excellence (like Walmart) and not heavily focused on product innovation as a primary business goal. In that kind of environment, there is typically some kind of Project Portfolio Management approach to determine how to allocate the company’s IT budget to get the most “bang for the buck”. The company needs to choose the projects that are going to provide the highest amount of leverage to the business and then monitor those efforts to see if they really do produce the desired effect.

At the level of a single team development process, the effort may look the same as a product development company, but the business context is entirely different and requires some adaptation. The primary difference is in the level of planning required. In a product development company, you might be able to use a highly adaptive approach with almost no Product Backlog defined in advance and you might be able to almost completely define the requirements for the project as it progresses looking no more than 2-3 sprints ahead into the future. That’s what a pure Agile approach looks like; however, that doesn’t work in environments where there is more of a need to plan the project in advance and estimate the schedule and costs of the project to support a higher-level project portfolio management decision process.

In those environments, there is a typically a need to at least define the Product Backlog to a sufficient level to define the scope of the project and to provide at least a high-level estimate of the costs and schedule for the project. Some people will say that it is futile to attempt to estimate the costs and schedule for an Agile project because the requirements are only going to change. I don’t totally buy that…this is not an all-or-nothing decision to have a project totally planned in detail (like the Waterfall approach) or not planned at all (totally adaptive). There are plenty of alternatives between those two extremes to pick a level of planning that is appropriate for the project, the level of uncertainty in the requirements, and the business environment that it has to operate in.

Some people will also say that you have to force the entire company to change its culture to become totally Agile in order to implement an Agile development process. A company has to define its culture around what makes the most sense for the business it operates in and that may or may not align very well with an Agile development process. In the product development company, it probably aligns very well and it makes sense for the company to become more Agile in delivering new and innovative product features to market quickly. In a company like Wal Mart whose business is leveraged primarily by operational excellence and lower costs, that is probably not the case and you can’t necessarily force the company to adopt a totally Agile culture.

The key message is that you have to consider an Agile development process in the context of the business it operates in and many times you need to adapt an Agile development process to fit the business environment. Within the development process itself, the process may be largely the same – the differences may come into play at a higher-level such as the project portfolio management layer that wraps around the project. Those higher-level layers could also be Agile (Dean Leffingwell’s Scaled Agile Framework is an example), but that kind of complete, top-to-bottom Agile transformation just doesn’t work in all business environments.

The Value of Estimation in an Agile Project

I participated in a discussion recently on the subject of Agile Estimation – the individual who started the discussion indicated that his team was not very good at estimating and asked whether it was important for them to become more proficient in estimating the level of effort required. I think it is very important and it is a skill that is often neglected in Agile development projects. Here’s why I think it is important:

  • At a project level, there is a need for some kind of planning to estimate the scope of the effort and to set expectations of how long it is going to take to finish the project. Very few projects are given a “blank check” to get something done without some kind of expectations associated with the cost and schedule of the project and it’s irresponsible to not set and manage those expectations. I have seen Agile projects where the project has gone on-and-on for an extended period of time without a plan for when it would finish; and, in one case, the project was so large that it couldn’t possibly be done by a single Agile team and that wasn’t discovered until well into the project when the project had to be re-planned and estimated.
  • At a more tactical level within a project, there is an ongoing need for the Product Owner to evaluate the value produced by stories against the level of effort required to develop that capability to ensure that the work is being prioritized properly to maximize the value the project produces. If you only know the business value to be produced without an idea of the level of effort associated with it, it is very difficult to make a good decision about that.
  • There is also a need to accurately size the level of effort that can be taken into a sprint so that it can be completed successfully. It can be demoralizing for a team to never finish a sprint successfully because they weren’t able to accurately estimate the level of effort required.

There is no question that estimation is a difficult thing to do in an Agile environment, but the importance of doing estimation is not well-understood and developers sometimes resist making estimates. The important thing is to define the approach for doing estimation in the context of the project you’re operating in. Some projects may have very uncertain requirements and may be very difficult to estimate and that may be considered OK but that doesn’t have to be the norm for all Agile projects. It is not an all-or-nothing decision to be totally adaptive with no plan or estimates at all or to be rigidly plan-driven. There are plenty of alternatives between those two extremes.

Methodology Myopia

Have you ever met someone who has “Methodology Myopia”? The symptoms of this problem are that the person thinks that there is one particular methodology (whatever it might be – Agile, Waterfall, or something else) that is a universal solution for any kind of project you might have. This “one size fits all” approach many times results in people force-fitting a project to a methodology whether it fits or not and practicing that methodology rigidly without necessarily tailoring it to fit the situation. I believe that a better approach is to fit the methodology to the project and sometimes that might require blending elements of different methodologies together to fit the situation.

Why does this problem exist?

  • Sometimes people only know one methodology. For example, for many years, project managers were heavily schooled in the Waterfall approach and there was no need to learn any other methodology.
  • It takes a lot more skill to know how to tailor a methodology and/or mix-and-match elements of different methodologies to fit a situation. It requires knowledge of a broader range of methodologies and a deeper understanding of the principles behind the methodology rather than just the mechanics of how it is implemented.
  • Sometimes people get so over-zealous about a particular methodology that they lose their objectivity and forget that any methodology has limitations and needs to be applied intelligently and tailored as necessary to fit a situation. For example, there are some agilists who are so zealous about Scrum that they completely reject any kind of more traditional plan-driven approach as obsolete and no longer relevant.
  • Consultants often fan the flames of this fire by building the frenzy about a particular methodology or approach being the best thing possible because they are selling their services based on a particular approach or methodology.

In my book, I use the analogy of a Project Manager as a “Cook” versus a Project Manager as a “Chef” (credit for this original idea is due to Bob Wysocki). A “Cook” knows how to prepare a limited number of recipes and do it by the book – a “Chef” knows how to prepare a much broader range of recipes with different and more exotic ingredients and also knows how to create entirely new recipes when required. This is a major challenge for the Project Management profession today…we need more Project Managers who are “Chefs”, who know how fit a methodology or combination of methodologies to a given situation. Many times this will require learning how to blend seemingly disparate approaches like Agile and plan-driven methodologies in the right proportions to fit the situation. Any Project Manager who only knows how to do a traditional plan-driven project management approach based on Waterfall may be severely limited in their career options in the future.

This is a difficult challenge to take on because I don’t believe anyone has all the answers of how to go about blending Agile and traditional plan-driven principles and practices together in the right proportions to fit a given situation. That takes a lot of skill, there is no “canned” solution, and the knowledge of how to do that as well as what works and what doesn’t work in different environments is constantly evolving at this time. However, even though this is a difficult challenge to take on, it’s important to recognize that this problem does exists and begin to take action to resolve it.

For many Project Managers, this requires developing a very broad and deep knowledge of both Agile and traditional plan-driven project management approaches as well as some actual experience in to blending them together that goes well beyond what is covered in the PMI-ACP certification. And, for both Project Managers and agilists, it requires getting past many of the stereotypes, myths, and misconceptions that have polarized these two communities to begin to see how these seemingly disparate approaches can be complementary to each other rather than competitive with each other.

The Future Direction of Agile Project Management

I think there’s a lot of confusion in the project management profession about what the impact of Agile is on the career direction for a project manager. Some people seem to think that it is only a matter of getting another certification and I’ve participated in several LinkedIn discussions lately where project managers were asking questions like: “What certification should I get in order to get into Agile (CSM/PSM, CSPO, or ACP)?” I think that the impact is much broader and more significant than that and its not as simple as just getting another certification; it requires rethinking some of the fundamental tenets of project management that have been practiced for years and it may cause many project managers to redefine their career direction.

The answer to the question of “what certification should I get” depends on what role you want to play and it requires some thought and planning because there is no well-defined role for a project manager in Agile at the team level. There are several possible career directions for project managers with regard to Agile.

  1. Become a ScrumMaster or a Product Owner – These two roles are both completely different from a normal project manager role and require some very different skills. You may not have to completely throw away your project management skills, but you would have to rethink them considerably and you may not use them very fully at all.


    A ScrumMaster is what’s known as a “servant leader”. The Scrum Alliance defines the primary responsibilities of a ScrumMaster as follows:

    • Ensures that the team is fully functional and productive
    • Enables close cooperation across all roles and functions
    • Removes barriers
    • Shields the team from external interferences
    • Ensures that the process is followed, including issuing invitations to daily scrums, sprint reviews, and sprint planning
    • Facilitates the daily scrums

    There’s a few project management skills that might be useful (at least indirectly) for that role but it doesn’t utilize much of the planning and management skills that a project manager typically has. Becoming a ScrumMaster may not make sense as a career direction for many project managers.

    The Scrum Alliance defines the primary responsibilities of a Product Owner as follows:

    • The product owner decides what will be built and in which order
    • Defines the features of the product or desired outcomes of the project
    • Chooses release date and content
    • Ensures profitability (ROI)
    • Prioritizes features/outcomes according to market value
    • Adjusts features/outcomes and priority as needed
    • Accepts or rejects work results
    • Facilitates scrum planning ceremony

    Again, there are a few project management skills that might be useful to perform that role; but the Product Owner role is much more similar to a Product Manager than a Project Manager, it would take the project manager in a completely different career direction, and also may not make sense for many project managers.

  2. Project/Program Management of Large, Complex Enterprise-level Agile Projects – There is a legitimate role for project managers in managing large, complex enterprise-level projects; however, there are several things to consider about planning your career in that direction:
    • This role is limited to large, complex projects that typically require multiple Agile teams and require blending together some level of traditional plan-driven and Agile principles and practices in the right proportions to fit the situation. This role doesn’t exist at all on most small, single-team Agile projects.
    • This role requires some very significant skills that can be very difficult to attain. Many people may assume that the PMI-ACP certification qualifies you to perform this role. It is a step in the right direction, but a lot more experience and knowledge is needed to perform this role including:
      • Knowing how to blend traditional, plan-driven principles and practices in the right proportions to fit a given project,
      • Adapting an agile approach to fit a business environment, and
      • Scaling Agile to an enterprise level.
      • You have to be kind of a “rock star” Agile project manager to perform this role.

      • This role is very new and evolving, not well-understood, and not necessarily well-accepted. At this point in time, I think any project manager who takes on a role like this has to be somewhat of a pioneer in taking on an enormous challenge to prove the value of project management in this kind of environment.

These are difficult choices for a Project Manager to make, but the key message for any project manager should be that Agile will force them to make some significant choices about their career direction and it isn’t as simple as just going out and getting another certification (like ACP). In many industries and application areas the project management role associated with small, single-team projects may be completely eliminated by Agile. There may be some project managers who are not significantly impacted by this such as project managers in the construction industry, but even in those industries some knowledge of Agile principles and practices may be essential.

My motivation in publishing my Managed Agile Development book has been to help project managers understand this impact and hopefully to help them make the right decisions to proactively move their careers in the right direction. I’ve also begun working with Boston University on developing a course on Agile Project Management that will also be designed to address this need.

The Agile Project Management Pendulum

The original Agile movement started out as a revolution against the traditional Waterfall methodology which was viewed as very cumbersome, bureaucratic, and inflexible. The need for that revolution was absolutely correct – an Agile approach does offer many advantages where a more adaptive approach is needed; particularly in environments where the requirements are very uncertain and subject to change. However, as in many other revolutions, there’s often a tendency for the pendulum to swing too far in the opposite direction to make a correction.

Pendulum

In particular, a lot of polarization has developed between some people in the Agile community and people in the more traditional project management community.

  • There are many agilists who are entrenched in their perspective that the only way to be “agile” is strictly “by the book” and that there is no need for project management at all – they see project management as a role rather than a set of principles that can be adapted to a broad range of different environments, just as the agile principles can also be applied to a broad range of different environments
  • There are some project managers who are equally entrenched in thinking that traditional, plan-driven, control-oriented approaches are the only way to do project management and have not learned how to integrate an Agile approach into their overall toolkit

The pendulum has begun to swing back towards the middle a bit and there’s less polarization today than there was several years ago, but some of that bias still does exist on both sides of this fence. Some of the progress that has been made over the past few years has been:

  • PMI has recognized the need for integrating an Agile approach with a traditional project management approach and has begun moving in that direction with the PMI-ACP (Agile Certified Practitioner) certification. Although it is a step in the right direction, it doesn’t go far enough, in my opinion. It doesn’t really address the larger question of how a project manager would go about blending together Agile and traditional plan-driven principles and practices in a real-world situation and what role a Project Manager would play in an Agile project to use this knowledge.

    Is the PMI-ACP certification PMI’s answer to the Agile CSM certification? That would imply that the goal of the PMI-ACP exam would be to compete with CSM and train project managers for the Scrum Master role and I don’t believe that makes sense at all. The only way it makes sense, in my opinion, is for a Project Manager to take on a higher-level role in larger projects that require blending together some traditional plan-driven and Agile principles and practices in the right proportions to fit the situation, but that role is somewhat undefined at this point and also not necessarily widely understood and accepted.

  • As Agile begins to be utilized for larger and more complex, enterprise-level projects; there is an increased recognition in the Agile community that an Agile development process like Scrum that works very well at the team level doesn’t necessarily scale very well without some kind of overall management framework and several different frameworks have been developed to fill this need.
    1. The Scaled Agile Framework developed by Dean Leffingwell is an example of a relatively complete approach that incorporates higher levels of project and program management as well as project portfolio management into an overall framework that is fairly Agile from top-to-bottom; however, it is not easy to implement and would typically require a very major transformation for a company to adopt that kind of approach.
    2. For companies who want to integrate an Agile development approach at the team level into a more traditional management framework, the Managed Agile Development approach defined in my latest book and Scott Ambler’s Disciplined Agile Delivery framework are both alternatives that can be used in a more traditional management environment.

It’s time to get past the polarization that has existed in the past and begin to see Agile and plan-driven approaches as complementary to each other, rather than competitive. It’s not an “either-or”, “black-and-white” alternative to adopt an Agile or Waterfall approach as some people have portrayed it; it’s more of a continuous spectrum of alternatives offering different levels of control and adaptivity as needed to fit a given situation. That’s the challenge I’ve tried to take on in my two books on this subject.