Learning to Live with Uncertainty with Agile

Effective management of uncertainty and risks is probably the single-most important factor in the success of any project (Agile or not). Prior to Agile, there may have been a false sense of security from thinking that you could completely eliminate or significantly reduce uncertainty by developing a very detailed requirements document for a project upfront and then managing changes against those requirements to control the scope and direction of the project as the project was in progress. That sort of approach is often taken in order to gain some level of predictability over the costs and schedule of a project; however, it turns out in many cases that a good deal of that feeling of security may be an illusion and it is very difficult to accurately predict all the requirements, costs, and schedules of a software development project.

With Agile, in many situations, the pendulum has swung almost completely in the opposite direction – a lot of people in the Agile community seem to feel that any attempt to try to reduce uncertainty upfront in order to estimate the costs and schedules of a project is totally futile because there might be a lot of uncertainty involved and the requirements are only going to change anyway. For that reason, there is a tendency to not do any significant amount of upfront planning at all. I think that’s an over-reaction and it can lead to significant problems.

Here’s an example of the impact that it can have…I was recently involved in helping to rescue an Agile project that had gone on for about two years with no end in sight. One of the basic problems was that the project had started without a sufficient level of upfront planning to assess the scope of the effort and it was just assumed that it could be completed in some reasonable amount of time by a single agile team. After about two years into the effort, we stepped back and did some re-planning and found that the scope of the effort was much larger than a single Agile team could accomplish in any reasonable amount of time. If more upfront planning had been done prior to the start of the project, perhaps that probably would have been discovered much earlier.

There are two primary problems I have often seen in this area:

  1. Analysis Paralysis – Delaying making commitments or delaying making any decision at all until the information required to make the decision has a very high level of certainty about it and all of the risks associated with a project are well-known and completely understood. This is often associated with the traditional, “Waterfall” style of project management. Unfortunately, this mode of operation has been the norm in some companies that are very control-oriented and insist on accurately knowing the cost and schedule of a project before making a significant commitment to it.
  2. Cavalier Approach – The opposite of that can be to take a very “cavalier” approach to not worrying about managing uncertainty and risk at all, just start doing a project with very little or no upfront planning, and assume that whatever uncertainty and risk is inherent in the project will be discovered and somehow work itself out as the project proceeds. This is the approach I’ve commonly seen on a number of Agile projects.

Neither of those approaches is ideal and each can lead to significant problems – a better approach is to make informed decisions as best you can based on the level of risk and uncertainty in the situation. Very few decisions in life are based on either 100% known or 100% unknown information and learning to deal with uncertainty is a skill that any good Project Manager learns over time. That skill is still very important in an Agile environment and shouldn’t be ignored. (In other words, don’t throw the baby out with the bath water when you convert from a traditional project management approach to a more Agile approach)

You often have to make informed decisions based on very uncertain information because waiting for the information to become more certain might significantly delay the project or might not make sense at all. The approach that seems to work best is to separate the “known” from the “unknown”, make some hypotheses or assumptions about the unknowns if necessary, and then evaluate the risks of going forward based on those assumptions or hypotheses. If you do decide to go forward, you shouldn’t forget that those assumptions were only assumptions and may need to be revalidated later on. (Many people often forget to do that) In a traditional project management environment, a risk register or something equivalent to that might be used for that purpose – that level of formality may or may not be necessary in an Agile environment, but the principle is nonetheless important.

There’s an art and a skill associated with making decisions in an uncertain environment. Anyone who has done any gambling where you have to bet on the outcome of an unknown event (like what card you’re going to be dealt next) will understand that. Here’s an example: most people are familiar with the game of “Blackjack” where the players play against the dealer and both try to win by getting the highest possible hand without going over 21. Suppose I have been dealt two cards and have a total of 16 points in my hand and the dealer is showing one card as a “10” with the second card face down. There are several very significant things I don’t know:

  1. I don’t know what the dealer’s second card is because it is turned face-down
  2. I don’t know what card I might be dealt next

Both of those are significant unknown’s but I have to make a decision anyway. I can make an assumption that the dealer’s second card is either a “10” or a face card (Jack, Queen, or King) for a total of 20 points. That assumption seems to make a lot of sense because there are more ways for that to happen (Ten, Jack, Queen, or King) than any other individual card. If I make that assumption, I know that my 16-point hand is going to lose against his 20-point hand and the best course of action would be for me to take a hit and risk going over 21 because I’m going to lose anyway if I don’t take a card or I don’t do anything at all.

That’s an example of an informed decision – I separated the known’s from the unknown’s and then made a reasonable assumption about the unkown’s based on the risks I saw in the situation. The alternative would have been to make no decision at all because the unknown’s were too significant or to make a “shoot from the hip”, uninformed decision to do something without doing any real analysis or really thinking it through at all and without taking advantage of the facts I do know in order to make a decision. I think that one of the most important factors for doing that effectively is to make an honest and objective assessment of the situation. There are several errors I have seen often with this:

  1. Overestimate the level of confidence in what is known and make a poor decision based on incomplete information
  2. Make an assumption about incomplete information, make a decision based on that assumption, press forward, and then forget that it was an assumption and never retest or revalidate it later
  3. Ignore whatever known information is available for making a decision and just proceed forward based on an uninformed decision without taking advantage of what is known

Is Agile Project Management an Oxymoron?

A few years ago, I attended a presentation by a well-known agilist who said that “Agile Project Management is an Oxymoron”. That kind of perception is based on a lot of stereotypes about Project Management that need to be changed. Here are some of those stereotypes that exist:

  1. Project managers are very command-and-control oriented
  2. Project managers are heavily associated with plan-driven “Waterfall” processes and document-centric methodologies where you need a plan for almost every aspect of any project and you also need a document, checklist, or a form for almost anything you do; and there is no role for a Project Manager in an Agile environment that requires a much more adaptive approach

There are certainly project managers who have some of those characteristics, but it’s unfair to make a judgment that all project managers are that way or to write-off the whole project management profession as being irrelevant to Agile because some project managers may be that way. However, we in the project management profession need to recognize the impact of Agile and recognize that a shift in thinking is needed to broaden our perspective on project management to correct these stereotypes that do exist. Here are some of those shifts in thinking that I believe need to take place.

  • Command-and-Control Orientation

Project managers are noted for getting things done and as a Project Manager, I’ve been in numerous situations where a customer has expected me to “ramrod” a project to get it done to meet time and budget constraints. That’s the way Project Managers have operated for many years and it naturally leads to a very assertive role where the Project Manager takes total responsibility for the success of the project and drives everyone on the project team to deliver results to achieve success. Both project managers and their companies need to recognize the limitations in that approach.

The Project Management profession needs to go through some changes that I saw the Quality Management profession go through a long time ago. In the early 1990’s I worked in the Quality Management profession. What I learned at that time was that it is self-defeating for a Quality Manager to take on too much ownership for quality. An effective Quality Manager has to be somewhat of a “missionary” and engage others in the effort to own responsibility for “quality” to be successful. If the only people who are responsible for “quality” are the people in the Quality Management department or the Quality Assurance department, it’s not going to be very effective.

The same thing also applies to project management. The project management responsibility for an Agile project should be shared by everyone on the team – even a developer has to be a little bit of a project manager to plan, organize, and manage the tasks that he/she is responsible for. Operating in this environment calls for more leadership than management – a leader sets the vision as well as goals and objectives to be accomplished; puts in place the people, process, and tools to get it done; and then steps out of the way as much as possible.

That approach is counter-cultural for many project managers because project managers are noted for getting things done and many companies expect that of their project managers. So it requires a shift in thinking in both project managers and the companies who manage them to bring about this change in orientation. There is also a natural resistance to this in some instances because if you do it successfully and really empower the teams you manage, you can eliminate the need for a project manager and put yourself out of a job and that might be the right thing to do – there is no defined role for a Project Manage in an Agile team that is working as it should be. The key thing is that it requires a change in thinking in project managers to let go of that role that they may have played in the past and move up to playing a higher-level role that provides more value-added. It’s ironic, that as I think back on some of the recent projects I’ve managed, if I’m asked what I did to make it successful, the answer is “I did as little as possible”.

  • Association with “Waterfall” Methodology

    Project managers are also heavily associated with plan-driven, document-centric approaches like the “Waterfall” methodology that can be extremely cumbersome and bureaucratic and there is no need for a project manager in an Agile environment. Unfortunately, I think this stereotype is somewhat legitimate and I know a number of project managers who are still holding on to a traditional PMBOK-style, plan-driven, document-centric, “Waterfall” approach to project management. I think there are a number of possible reasons for that:

    • Job Security – Many project managers have been heavily trained in traditional project management skills and PMBOK and it’s a big risk to give that up because that’s what their knowledge is based on and that’s the value-added that they have been trained to deliver. The role of an Agile Project Manager is also very undefined and uncertain at this point and might require a significant amount of retraining. It’s understandable that some project managers might not feel comfortable making that leap of faith to re-orient their career around Agile Project Management
    • Binary, All-or-nothing Thinking – A lot of people on both sides of the fence have the perception that it is a binary choice between a totally plan-driven and heavily controlled approach (e.g. “Waterfall” and a totally unplanned approach with no control (e.g. “Agile”) and that is not the case. I’ve previously written a couple of articles on that subject:

      Many project managers and many companies do not see that there is a lot of middle ground between extreme Waterfall and extreme Agile…those alternatives may not be well-defined and it can take more skill to figure out how to develop an approach that blends the right proportion of Agile and traditional project management principles and practices to fit the situation but it definitely can be done. Unfortunately, many people make the mistake of force-fitting their business and their projects to one of these extremes rather than fitting the methodology (or combination of methodologies) to their business and to each particular project.

What is the solution to this? This is basically a change management problem and I’ve learned over the years that there are three elements that are essential to any significant change management initiative:

  1. “Burning Platform” – There has to be a sufficient level of “pain” associated with the current situation to recognize that it is untenable and making a change is crucial. People have to get past the “denial” stage and recognize that Agile has a profound impact on the project management profession and in the not-too-distant future, project managers who only know how to do traditional project management PMBOK-style Project Management and don’t know how to take a more Agile and adaptive approach when it makes sense are going to be at a serious disadvantage in the job market.
  2. Vision for the Future – The next step in any successful change management initiative is that there has to be a vision for what the future looks like after the change has taken place. To do that, we need to better define what an Agile Project Manager is and what role he/she will play. I’ve tried to help develop that vision with the two books I’ve published but there’s still work to be done in that area to better define that vision. I’m currently working on developing a graduate-level course on Agile Project Management with a major university in the Boston area that I’m hoping will also help fill this need.
  3. Progress in the Right Direction – In any change management initiative, there will always be some hold-outs and laggards who will wait on the sidelines to see if the change is going to be successful or not before jumping on board. I’m sure that there are a number of project managers who don’t want to take the risk of becoming Agile Project Managers until that role is much more well-defined and proven and they will wait until more people have actually demonstrated success in that role.

That will obviously take time and isn’t going to happen overnight, but the first step is to recognize the problem. I hope this blog post helps others see this problem as I do. I welcome your comments and thoughts on this!

Was Steve Jobs an Effective Agile Leader?

I watched the movie “Jobs” this weekend about the life of Steve Jobs and his career at Apple and it was very thought-provoking.  Steve Jobs was absolutely brilliant, embodied a lot of Agile values, and he was enormously successful in developing some very innovative products in a relatively short amount of time that made Apple very successful;  but he was ruthless, tyrannical, and very insensitive in his relationships with people.  I was thinking – is that style of leadership really consistent with Agile and is it an effective style of leadership for an Agile environment?

  • Much of the thinking behind Agile is based on the idea of empowered and self-organizing teams where the product definition bubbles up rather than being driven down from above,  Steve Jobs’ leadership style doesn’t seem to be very consistent with that model, but he was very successful in getting things done.
  • Another thing that seems to be not entirely consistent with Agile is that Agile is based on the idea of teams working at a “sustainable pace” and it was apparent that many of the teams that worked under Steve’s direction at Apple worked incredible hours to get things done but they were very passionate and dedicated to their work.

Here are some quotes from Steve Jobs that indicate his values that are related to Agile:

  • Focus and Simplicity – “People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying no to 1,000 things…
    “That’s been one of my mantras – focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it’s worth it in the end because once you get there, you can move mountains.”
  • Planning – “Again, you can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future. You have to trust in something – your gut, destiny, life, karma, whatever. This approach has never let me down, and it has made all the difference in my life.”
  • Quality – “Be a yardstick of quality. Some people aren’t used to an environment where excellence is expected.”
  • Innovation – “Sometimes when you innovate, you make mistakes. It is best to admit them quickly, and get on with improving your other innovations.”
  • Requirements and Customer Needs – “You can’t just ask customers what they want and then try to give that to them. By the time you get it built, they’ll want something new
  • Seeing the “Big Picture” – “A lot of people in our industry haven’t had very diverse experiences. So they don’t have enough dots to connect, and they end up with very linear solutions without a broad perspective on the problem. The broader one’s understanding of the human experience, the better design we will have…To turn really interesting ideas and fledgling technologies into a company that can continue to innovate for years, it requires a lot of disciplines.”
  • Continuous Improvement – “I have a great respect for incremental improvement, and I’ve done that sort of thing in my life, but I’ve always been attracted to the more revolutionary changes. I don’t know why. Because they’re harder. They’re much more stressful emotionally. And you usually go through a period where everybody tells you that you’ve completely failed.”
  • Tools – “It’s not the tools that you have faith in – tools are just tools. They work, or they don’t work. It’s people you have faith in or not.”
  • Leadership Style – “It’s not about charisma and personality, it’s about results and products and those very bedrock things that are why people at Apple and outside of Apple are getting more excited about the company and what Apple stands for and what its potential is to contribute to the industry…The people who are doing the work are the moving force behind the Macintosh. My job is to create a space for them, to clear out the rest of the organization and keep it at bay.”

And one of his most famous quotes that really sums up his values is “Stay hungry, stay foolish”.  The source of the above quotes is:

http://www.brainyquote.com/quotes/authors/s/steve_jobs.html

My thoughts on this are that:

  • Steve Jobs definitely had some “rough edges” in his relationships with people but he embodies many of the characteristics of an effective Agile leader
  • There probably isn’t one leadership style that is effective in all situations and some “out of the box” thinking is definitely worthwhile rather than implementing some kind of “textbook” Agile approach in all situations

What do others think?