« October 2006 | Main | December 2006 »

November 29, 2006

Two Good links

A good description of using mini-projects within much larger projects to produce initial valuable results and learn from those results. There is a good case study of a 16 year World Bank project. Why Good Projects Fail Anyway. From the article:

The key is to inject into the overall plan a series of miniprojects—what we call rapid-results initiatives—each staffed with a team responsible for a version of the hoped-for overall result in miniature and each designed to deliver its result quickly.

An excellent article about the use of very short iterations to build a website. The Freedom of Fast Iterations: How Netflix Designs a Winning Web Site.

Don't forget: Agile Advice is offering a 10% discount on agile training courses in Toronto, Edmonton, Calgary, Ottawa, Vancouver and Halifax until Dec. 31st!

Posted by Mishkin Berteig at 12:00 PM | |

November 28, 2006

Planning vs. Commitment

In Scrum, there is some confusion around the various types of estimation that are done. Product Backlog Items are estimated and Sprint Backlog Tasks are also estimated. These two estimates, despite some similarities, are used for very different purposes.

Scrum advocates estimating the Product Backlog Items (Work Queue) using relative points. Each item is given a number of points that represents its size relative to the other items in the backlog. The team does this by identifying a small, well-understood backlog item and giving it, quite arbitrarily, 2 points of effort. All other items are estimated relative to this item.

For the Sprint Backlog Tasks, the estimation is usually done in "ideal hours". This is because tasks tend to be relatively fine-grained and in the context of developing software, the team typically breaks backlog items into tasks that take less than a couple of days of real effort. Again, there is a strong "relative" estimation component here where the team might look at tasks that are simpler and easy to implement and estimate those first, while the rest are estimated relative to those simple ones.

These two types of estimation are used for different purposes. Product backlog estimates in points are used for planning purposes. Task estimates in ideal hours are used for team commitment purposes.

Points -> Planning

When the project starts, an initial Product Backlog is created to be at least a skeleton of all the functionality that needs to be built. As described in a number of places, including both of Ken Schwaber's books, the items at the top of the Product Backlog are at a more detailed level (and therefore smaller in effort) than those items at the bottom of the backlog.

To plan out the project, the team must estimate all the backlog items using points. The team guesses how many items on the top of the backlog it can do in its first sprint. Then, taking drag factors into account, the Product Owner can take the number of points estimated for the first sprint and divide that into the total number of points in the backlog to find out how many sprints the work might take. As the team goes along, this can be redone based on the number of points completed in the previous sprint.

At no time are the points used by the team to determine how much work they will commit to in a sprint. The team's capacity for planning purposes is measured in points/sprint, but this is not used to constrain the team's commitment. Instead, the team uses its estimates at the Task level.

Digression: Bananas -> Commitment

"Bananas"? Yup, bananas. A different word than points, and a word that has no connection to actual effort. The problem with estimating in Ideal Hours is that people think that there should be some predictable relationship to real hours. This is both false and unnecessary. Tasks in the Sprint Backlog can also be estimated in relative units and since the work "points" is already used for Product Backlog Items, I would like to introduce "bananas".

The team estimates the tasks in relative bananas. Now, when the team completes its first sprint, it can measure how many bananas it estimated at the start of the sprint, and how many remained at the end of the sprint. The difference is the team's velocity which is to be used for commitment purposes. You can't use velocity for planning purposes because the team hasn't broken the whole Product Backlog into tasks, only the backlog items that it is doing in the current sprint.

Based on velocity, the team can make an informed commitment at the start of the next iteration: never take on more bananas than your velocity allows.

I have deliberately used the nonsense term "bananas" to talk about the units of effort for tasks. I could have used "ping pong balls", "elephants", or "nebulous units of time" (NUTs). The point is, that these units can never be converted directly into man-hours or any other unit that might accidentally be used for comparison or performance evaluation purposes. One team's velocity cannot be compared to another team's velocity. Even if both teams use "bananas", there are quite a number of factors that would prevent comparison: what tasks are used to baseline, the skills of the teams, the organizational and environmental obstacles, the definition of done, the mood and amount of sleep of the person working on the task, etc. etc.

Posted by Mishkin Berteig at 11:27 PM | |

November 26, 2006

Agile Training - Discount and New Courses

The new winter and spring schedule for courses is up. There are three new courses: Introduction to Agile Work (a one-day course), Agile Team Member training, and Agile Queue Master training (product management / customer). Up until Dec. 31st, Berteig Consulting Inc. is offering a 10% discount to people who sign up for courses through Agile Advice. Follow this link:
Discounted Agile Training

Most courses are in the Toronto area, but some are scheduled for Edmonton, Calgary, Halifax, Vancouver and Ottawa.

Posted by Mishkin Berteig at 10:41 PM | |

November 24, 2006

How to Avoid Context Switching

Given the huge interest in the article by Dmitri Zimine about context switching, and despite a couple of good articles about how to determine iteration length, there has been no empirical method described, only reasoning processes. This article describes a simple method to quickly determine iteration length by experimental means.

Step One

Start with an iteration length that feels right. Use the two articles below to try decide a reasonable length. In software development, this should be no longer that four weeks. In larger volunteer communities it might be as long as three months. In a work environment where you have to deliver daily, your iteration length might be two hours.

Step Two

Build your Work Queue, and plan your first iteration. Mark the tasks that come out of your iteration planning meeting so that you know that they are tasks that were planned. As you go through this first iteration, track all the interruptions you get by writing up a task for each interruption. Each interruption task should be marked so that you know it was an interruption. If you are using note cards on a visible task board, I like to have "normal" tasks on white cards and interruption tasks on fluorescent orange cards to help them stand out!

Step Three

At the end of your iteration, determine which interruption tasks could have been deferred. In other words, determine which were truly urgent and needed to be handled in the already short time of your iteration, and which could have been put on the Work Queue, prioritized, and therefore deferred to a later iteration. This will require collaborating with your Queue Master and possibly other stakeholders.

Step Four

Count how many non-deferrable interruption tasks your team had in the iteration. For this experimental method, you can assume that this is going to be your normal number of interruptions. Divide the length of your iteration by the number of non-deferrable interruptions. For example, if you had a ten day iteration, and two non-deferrable interruptions, you would have a result of five days. Also consider what was the maximum reasonable response time for these non-deferrable interruptions. The lower of these two values becomes your experimentally determined iteration length. But you are not quite done!

Step Five

Do it all again to verify your iteration length. Note that because of your shortened iteration length, you hopefully will have far fewer non-deferrable interruptions. After your second (shortened) iteration, you can adjust the iteration length shorter if necessary... but don't adjust longer (for now).

I've worked with enough teams to know that for a substantial portion of them, this method would result in very very short iterations. In the software world, a team is often asked to work on a project and support their previous project. This support work tends to mean dealing with various bugs in deployed software. This is one reason why I have become such a strong advocate that quality is not negotiable.

I worked with one team that did something similar to this method (although not as rigorously) and we decided that we needed to try an iteration length of two days. This was painful for the team, but they badly wanted to build trust with their stakeholders. Their interruptions were causing them to constantly fail on their commitments. By switching to these two day iterations, they were able to defer the bulk of their interruptions and meet the commitment they made as a team in the iteration planning meeting.

Articles about iteration length:

Mike Cohn's article: "Selecting the Right Iteration Length for Your Software Development Process"

Mishkin Berteig's article: "The Pros and Cons of Short Iterations"

Now that you have gotten to the end of the article, I can admit something to you: this article is badly named. It should be named "How to determine how often to context switch so that you can meet your commitments and build trust with your stakeholders."

What this article doesn't help with is the pain of context switching. The teams that I have see that use short iteration lengths all find ways of making context switching less painful. Automation, good tool choices, workspace arrangements, etc. all play a part in this. But there is no secret ingredient to make context switching painless. Sorry!

Posted by Mishkin Berteig at 07:55 AM | |

November 21, 2006

Obstacle Types

The Art of Obstacle Removal is an important thing for the Process Facilitator to get good at. I was working with a client today and realized that there are really three broad categories of obstacles and only one category is easy to see. The other two are much harder for people to identify and deal with.

Category One: Tangible Barriers

This is the easy type to identify. Physical obstacles such as distance or walls are the most obvious. And usually people have no trouble identifying other tangible barriers such as bureaucracy or bad habits.

Category Two: Obstacles of Absence

These types of obstacles are harder to identify because they often take creative thinking to see a possibility of improvement by adding something to the environment. Some general sub-categories include a lack of trust, a lack of automation and a lack of knowledge.

Category Three: Chaos

These is often the hardest type of obstacle to identify. Conflict between team members is hard to speak honestly about. Too much context switching is also hard to acknowledge because it often seems like it's a good thing to switch to something else - there's always some sort of reason.

Are there any broad categories that I've missed? Any good examples of these different types of obstacles?

Posted by Mishkin Berteig at 05:29 PM | |

November 17, 2006

Thinking/Doing Loops

Ryan Cooper, who recently attended one of my agile training courses said:

It doesn't make sense to separate the thinking from the doing.

So the real question is: why do we do this so often?

Levels of Separation

It is probably good to note that there are levels of separation. If you're in a big bureaucratic organization you will find hugely separated thinking and doing.

For example, at one large organization that I know well, there is a part of the organization dedicated to determining the best possible process to follow for all the different types of projects that are run: software projects, business projects, marketing projects, etc. The people in this process group sit together and think about all sorts of interesting things like how to meet regulatory compliance constraints, how to measure things really well, how to avoid most mistakes, how to make sure that there is a careful balance of responsibilities, how to make sure that everyone who cares about a project gets a say, and how to make sure that everyone who cares about a project can throw in their version of the kitchen sink. These people get around to interviewing the managers of the people who actually work in these projects... maybe once every six months.

The people "working" in the process group are almost completely separated from the reality of the people doing the work. The feedback cycle is at least twelve months. Let's say that a project starts out using Version N of the process defined by the process group. The project runs for six months. The process people interview the manager of the project and incorporate the feedback in Version N+1 of the process which is released into the wild six months later.

The trouble is that the twelve months for the feedback cycle is actually the least bad it could possibly be. More likely it looks like this: six month project? Too small, the feedback isn't terribly important. Two year, forty person project? That's more like it! Let's get lots of feedback in the post-mortem for this project! Project starts with Version N of the process. Thirty-six months later the project finishes (if lucky!). Post-mortem done. a few months later someone from the process group receives the post-mortem report (200 pages of boring, poorly written text). The process group decides that the $100 million worth of feedback deserves its own project. The process group launches a major process revision project based on the post-mortem report. One year later, the process group releases Version N+1 of the process. Total elapsed time for feedback: fifty+ months! Total waste of time: yes.

Okay, so that's a worst-case scenario.

There are lots of smaller ways that we separate thinking from doing.

Individual

The most direct pairing of thinking and doing happens at the level of the individual person. A person's own thoughts and creative efforts are directing their actions, and they are observing and learning from those actions to change future thoughts. This feedback loop is the easiest and most natural (although sometimes we forget to learn from the results of our actions). Sometimes, this feedback loop is so tight that we get into a state of flow, the "zone" and it seems as if our actions and our thoughts are perfectly at one.

This level is limited in that many thing cannot be done by the efforts of a single human being. So we move up a level.

Team

The team works closely together so that thinking and doing is separated at most by trusted high-bandwidth low-latency communications such as a few people working together at a whiteboard, or two people working at a computer together. All the members of the team have equal participation in the thinking/doing loop.

Again, it is possible for a team to get into the "zone" and become a high-performing or "real" team. People on teams like this often can't remember who thought of what great idea or who executed some particularly amazing piece of work. Individuals offer their best talents, skills, experiences and are receptive to the offerings of the other team members.

This level of separation of thinking/doing does present some challenges: it is more difficult to get into a state of flow as a team, it is easier for the work to be disrupted by the environment, and sometimes the communication breaks down.

Group or Organization

There are some rare organizations which maintain an incredibly productive state above the team level. Toyota is oft-cited, but there are others too such as the "Good to Great" companies.

More often than not, however, groups and organizations fall into various degenerative types of thinking/doing separation. In software development, this is typified by the handoffs between people doing requirements to different people doing architecture to different people doing analysis and detailed design to different people doing development to different people doing testing. And that's not even taking into account project managers, managers, marketing, sales and customers! Yikes! No wonder phase-based software development is so painful!

Obligatory Conclusion

Agile methods help tighten up the thinking/doing loop by requiring cross-functional teams with product (what) and process (how) and technique (work) skills and authority actually on the team.

P.S. This is an expansion on the Cycles of the Mind stuff.

Posted by Mishkin Berteig at 02:54 AM | |

November 16, 2006

Trust vs. Camaraderie

Ryan Cooper has written an excellent article called Trust vs. Camaraderie where he points out that many "teams" have established a feeling of Camaraderie, but not established trust... and that this is a problem.

Posted by Mishkin Berteig at 10:08 AM | |

November 15, 2006

Quality as a Corporate Asset

Ken Schwaber has a great talk about quality which supports and confirms my earlier article called Quality is Not Negotiable.

Posted by Mishkin Berteig at 04:05 PM | |

The Case for Context Switching

Recently, Dimitri Zimine wrote an excellent little story about context switching. Joel Spolsky writes in "From the 'You Call this Agile' Department":

Dmitri is only looking at one side of the cost/benefit equation. He's laid out a very convincing argument why Sarah should not interrupt her carefully planned two week iteration, but he hasn't even mentioned arguments for the other side: the important sale that will be lost.

Okay... I'll bite.

Let's look at that argument from the perspective of the sales person first since that is where Joel's concern starts:

The Sales Guy Perspective

"I need the 'ezhibal' feature now! Big bucks coming soon if we can get it now."

Let's suppose that this urgent email has come in somewhere near the start of our two week iteration. The normal response to this in Scrum is to push back a little. The ScrumMaster says something to the effect of: "Look if you wait 7 days we can put this on the top of the list for our next iteration."

First reaction, and it's a normal one, is for Sales Guy to freak out. I've actually heard people saying things like "You're going to lose your job over this! I'm getting the VP involved and he's not going to like it" and then they stalk off to find the big dog to come and bark at us. Anyway, let's pretend that the Sales Guy is willing to be reasonable and not instantly escalate the "problem". So what he actually says is: "Look, this is super important, it'll probably only take a few minutes for us to talk about it and then we can figure out how long it will take to fix. Let's just do a quick phone call and yadda, yadda, yadda, blah dee blah..."

Enough of the Sales Guy perspective.

Nowadays, if I'm in this situation, I do a value assessement. I tell the team to keep working on their plan, nothing's changed yet, and I sit down with Sales Guy and the person who's sponsoring the current work and we start a discussion about the options of which there are really only two that work in Scrum:

  1. Stay the Course
  2. Cancel the Iteration

First, let's talk about how we decide which option to take. Then we'll talk about why.

Deciding on the option is easy. You look at the value of the work currently being done and compare it to the value of the work that Sales Guy needs. You take into account probabilities. If Sales Guy doesn't have a signed contract, then it's hard to day if there's going to be any real revenue from the 'ezhibal' feature. Still, you might be able to do an assessment of the probabilities based on your level of trust and history with the client, etc. You also need to take into account the time value of money. Does delaying the current work have consequences for another client or stakeholder? What is the cost of those consequences.

This is a relatively simple cost/benefit analysis and comparison. If you're not comfortable with money and numbers and spreadsheets, you better get comfortable!

Okay, so we have a way of comparing the two bits of work. Now let's look at the two "allowed" responses and a third "bad" response.

Stay the Course

Turns out that the potential benefit of the stuff Sales Guy wants is not quite as high as the potential benefit of the stuff that Suzie Stakeholder prioritized for the current iteration. Well, that's easy. Put the request from Sales Guy in our prioritized list of work and get to it when there is an appropriate level of benefit relative to the other work.

Cancel the Iteration

The stuff Sales Guy wants is super-valuable. So let's get the whole team to stop what they are doing and everyone supports this very valuble work. Stopping the whole team is appropriate because obvioulsy, the stuff they're working on isn't as valuable. Oh, and because we treat a team as a unit in Scrum. And because the team needs to commit to work, not individuals. This isn't so obvious... more later.

Peel Sarah off to do the 'ezhibal' Feature

This is what normally happens, and in a "normal" non-agile environment, it's probably okay. In a non-agile environment, Sarah hasn't made any commitments (she's been forced to agree to dates and scope, etc., but she hasn't made a commitment that she is empowered to live up to). So if she goes off and does this one little thing, it probably will be just business as usual. In an agile environment, the team has made a commitment and doing this work this way invalidates the team's commitment.

Why do we do it this way? The main reason is around trust and commitment. Yup, it's that soft icky stuff that we're so incredibly bad at that customers think that bugs are normal, that management shoves the kitchen sink into projects in the frustrated hope that they'll get something out of the IT team at the end of the project. Sound familiar to anyone?

Anyway. An iteration is a commitment. It is a firm and solid commitment. The team as a group of smart and honorable people is making a definite commitment to the rest of the organization to get a certain amount of work done in a fixed amount of time. In return, management is agreeing to give the team every support in reaching this commitment. When a team is new at this, they might get it wrong. But having done this with dozens of teams now, I know that after a few tries, the team gets the hang of it and commits to appropriate amounts of work, and management provides appropriate levels of support.

This commitment is essential for developing trust. And anything that comes in the way of the team meeting its commitment is considered "BAD". An obstacle to do away with.

This is interesting, because Joel's second example is about defects. And I strongly agree that defects are "BAD" and need to be dealt with at a very high level of priority. The reason is simple: they prevent a team from meeting its commitment.

One team I was coaching was constantly bombarded by these types of it'll-just-take-a-few-minutes-need-it-asap requests. They had many stakeholders and very very limited resources to service these requests. They had several small projects that were taking literally years to do because they couldn't get enough concentrated time on any one thing. This was considered normal and good in their environment.

The trouble is, no one had really looked at the overall consequences. Everyone was doing local optimization. For us geeks, we all know that local optimization is something to be avoided if possible. We climb a peak only to discover that we have to climb back down a ways to get up to the higher peak we now see is next to this one. We climb up that one only to discover yet another higher peak even further along thus requiring us to climb down and up again... When really what we should have done is stepped back a ways, looked at the lay of the land and said, "hey, that peak over there is the highest of them all, let's go climb that one."

Scrum helps us avoid local optimization by forcing all feature requests for a team to be prioritized in a list of work, and by allowing the team to complete small pieces of work so it actually gets _something_ done that you can learn from.

Joel said:

Agile is not supposed to be about swapping out one set of bureaucratic, rigid procedures for another equally rigid set of procedures that still doesn't take customer's needs into account.

True enough! But it also demonstrates a serious lack of understanding about what is being done in Dmitri's example! First of all, without being agile at all, it is possible to switch from 18 month projects to two week projects. Both can be bureaucratic as you please (well, actually, there's only so much bureaucracy you can cram into two weeks and still get something done). The shorter projects will allow you to be much more responsive to customer needs... by definition!

So what happens when you add in all the other things that agile really is about? Transparency. Truthfulness. Creativity. Learning. Meta-Learning. Prioritization. Self-Organizating Teams. Eliminating Waste.

Well, first of what you get is something that's damn hard to do right. It goes against almost everything we've been taught to do: the extreme of heroics of the extreme of careful planning and process.

Secondly, what you get is something that needs safety zones and meta-rules. Like mutual, freely-given, team-to-stakeholders commitment. Like assuming positive intent.

And thirdly, what you get is an environment where not only is the business getting what it needs more than it used to, but also, the team likes working with the business, and the business likes working with the team.

I admit that the point Joel is making isn't too different. Yes: look at the costs and the benefits. But agile isn't quite about instantaneous responsiveness. That's a red herring and I'm suprised that Joel threw it's stinky carcass into the discussion. Agile is also about balancing that responsiveness with the overall view of value for the team and the organization. The tool for doing that is the prioritized list of work, not the email message from Sales Guy to Sarah.

Posted by Mishkin Berteig at 04:13 AM | |

November 14, 2006

Process Facilitator "Smells"

I have now trained over one hundred people in my Agile Project Managmenet / ScrumMaster Certification course. I'm starting to see and hear some of the results of this training. There are a couple specific "smells" that I have become aware of.

Fortunately, I've been able to provide coaching to some of the organizations that have sent people to my course. There are quite a number of good things that happen, but there are a couple of things that seem to be "natural" misunderstandings.

  1. Spectating
    I put a lot of emphasis on the idea of a self-organizing team in my course. There are a number of exercises, an hour-long section, and many other points during the course when it comes up. With all of this emphasis, it seems that a few people have come away from the course with an extremely hands-off approach to the Process Facilitator role (ScrumMaster/Agile Project Manager). I think this is a natural and probably good reaction to the heavy-handed command & control approach that these people come from. However, there are a few things that should be considered minimum levels of engagement (listed below).
  2. Problem Solving
    There is also a great deal of emphasis put in the course on removing obstacles. I have seen several cases where it becomes the habit of the Process Facilitator to start solving every problem. This can happen in day-to-day work, and also in the retrospectives. Again, this seems to be a natural consequence of the desire to get in there and be of value. However, if the Process Facilitator writes down all the "things that need impovement" from the retrospective and then says "Okay! I'll take care of these things." then you know that the Process Facilitator has gone too far.

Appropriate Process Facilitator Engagement

Here are a few ideas on an appropriate level of engagement. Finding the right balance of enagement is not easy and there is no exact formula to follow. Partly it depends on your personality and skills as a Process Facilitator, partly it depends on the capabilities of the team, and partly it depends on the constraints of your work environment. Nevertheless, there are some types of engagement that you can persue with confidence. Here are a few concrete guidelines:

Posted by Mishkin Berteig at 01:49 PM | |

November 12, 2006

More on Scaling Agile

One problem with having multiple teams working on the same project will be the tendency to compare the teams' performance. Why is this a problem?

Why Not Compare Team Performance?

One of the main reasons is that the teams need to be collaborating not competing. There can be a small amount of friendly competition that might come naturally, but as soon as management starts paying attention to differences in team performance, the competition starts to get serious.

In the case of multiple teams working on the same project, the goal should be to maximize overall performance, not individual team performance. Too much competition can lead to unintentional or deliberate sabotage: failed communication, incomplete communication and downright dishonesty.

Motivating Teams without Comparing Them!

As Mary Poppendieck has written, measure up [pdf]. In a single-team situation this means that individuals are measured and rewarded based on team performance (their sphere of influence). In a multi-team environment, that means that the group of teams should be measured as a group and compensated as a group. This will encourage all teams to work towards the success of the overall project. I personally believe there is some room for individual-based compensation, but the way it is handled needs to be done so that it does not encourage sub-optimal behavior.

As well, each team can keep track of their velocity. Some coaches recommend using "ideal hours" or some other units to determine velocity (velocity = estimated work remaining completed / iteration). The trouble with doing this with multiple teams is that there will be a very real tendency to want to compare each team's velocity. Since velocity is a useful measure for team capacity, it is important to still have a way to measure it. One simple way to do this to prevent comparison is to use different units for each team. Team One might be measuring velocity in Estimated Ping Pong Balls Completed / Iteration... Team Two in Estimated Bananas Completed / Iteration... Team Three in Estimated Bogo-MIPS Completed / Iteration... etc. etc.

Motivating Collaboration

First off, management must make visible commitments to eliminating barriers to collaboration. For example, it is a great sign of commitment to re-organize office spaces so that all the teams are sitting near to each other. Every time the Process Facilitator identifies an obstacle that relates to collaboration (tools, process, physical environment, etc.) management should get right on it and fix it.

An ongoing investment in team-building training, workshops and exercises is also helpful. This type of investment helps people gain the skills necessary to work effectively with other people. Again, individuals need to see and believe that management cares about and values teams.

Finally, one of my pet peeves: when a project is over, keep the team together! Do not break them up and redistribute your "resources" to other efforts. The value of those people working together is substantial. The value of those people working together as a high-performance team is incredible! In a multi-team situation, it is reasonable to take teams from the overall group and re-distribute their efforts... but just don't break up the individual teams.

Miscellaneous Notes on Scaling Agile:

Twelve is still the maximum recommended size for a single agile team. Seven is really the sweet spot. A team larger than twelve people just takes too long to get into the Performing stage of team development. If you feel like your project needs more than twelve people actively involved, then you probably want to split into two or more teams... and then you have "scaled" agile.

If you have three teams of five people (or some similar configuration of people just over the 12 person limit), then they will work as a very close-knit group and a lot of the time will act as if they were a single team. They will probably plan iterations and do demos and retrospectives together.

Twelve teams working on the same project at once is about the maximum number before communication overhead is slowing everyone down too much. This is largely a factor of trust: with 150 or fewer people involved in an effort, it is possible for everyone to know everyone. More than that many people and it is no longer possible. Trust is just not an option anymore and bureaucratic controls take over.

If for some reason you need to do something in a small amount of time and you think it's going to take more than twelve teams of twelve people...? Break the effort into smaller chunks. Divide and conquer. Division can be across functional areas, structural areas or time.

Although I have heard of agile methods being used with groups larger than this, I haven't heard any success stories :-)

Check out my earlier introductory article on Scaling Agile in a large project situation.

Dean Leffingwell has an article about practices needed for scaled-up efforts at the Agile Journal. I glanced through it, but I admit that after I disagreed with his very first point (Intentional Architecture), I started to pay less attention.

He claims that refactoring of large systems is not possible (or at least infeasible). The odd thing is, most large projects that I have been involved with are being done exactly because an old system is not refactorable. A large telecom system, a large insurance system, a large data warehouse and a large GIS system are all being done with scaled up agile methods exactly because the old systems that are currently in place have become ossified to the point where they must be replaced.

These old systems were originally built with phase-based development approaches. At some point, people stopped refactoring because they were not given the space to do so. This drop in code quality turned into technical debt. The technical debt accumulated to the point where it was unbearable (maintenance costs, cost of change, etc.).

The problem with intentional architecture is that it goes back to the old assumption that you can do a good design for a system without the constant feedback from review, deployment and use done on a very short cycle time. Over and over, we are faced with the painful consequences of this attitude, and that is one of the key reasons we started to work with agile methods in the first place.

Martin Fowler makes a good case that scaling agile is the last thing you should do. I don't disagree! Scale your agile teams at your own risk!

It's nice for me to be able to say that I've worked on some agile projects over $10,000,000 in size, but the fact is that the cost could have been reduced substantially if the team size was lowered and the deadline extended. It is (relatively) simple to do a cost/benefit analysis of cross-team-coordination-overhead vs. the time value of early delivery of more functionality... although I've never seen anyone do it! If you know of an example of an organization doing this in a realistic way, I'd love to hear about it!

Are there other ways of supporting cross-team collaboration that you have seen?

Posted by Mishkin Berteig at 09:13 AM | |

November 11, 2006

Stronger Teams Blog

The blog Stronger Teams by Blaine Collins has some incredible stuff on it. I've read quite a few of the entries and they all show the importance and relevance of teams and teamwork to Agile Work methods. Considering some of my recent comments, thinking and coaching experiences, I love the current top article: Treat team members as ends and not means. This correlates strongly to what I consider a distinguishing feature of agile methods: meta-learning done by the team, and it also works well with the Agile Axioms.

Posted by Mishkin Berteig at 06:20 PM | |

November 10, 2006

How Two Hours Can Waste Two Weeks

Your developers have just planned a two week iteration. The next day Sarah continues her work on the completion of an important New Project. And here it comes - Urgent Stuff...

Your developers have just planned a two week iteration. The next day Sarah continues her work on the completion of an important New Project. And here it comes - Urgent Stuff:

>From: Project Manager
>Sent: 2nd Day Of Iteration
>To: Sarah, the Developer
>Cc: Yo, Dev Manager
>Prio: HIGH
>Subject: Coarechon needs Ezhibal
>
> I need you guys to find out if we can add "ezhibal" functionality to our old "Tounaf" project, and how much time this fix is going to take. We have a hot deal with an important customer who is almost ready to buy.

> Sarah please investigate ASAP. We can arrange a phone call to clarify requirements.

Soon there is a dialog that might go like this:

Project Manager: "Sarah, how long do you think it takes to check if it is feasible to add this 'ezhibal' feature?"
Programmer: "Well, I don't remember the old project details, but I can spend 2 hours to see if I can come up with something."
Development manager: "Sarah has her iteration planned. Why don't we stick to the plan?"
Project Manager: "What's the problem? It just takes 2 hours! What harm can it do?"

"It just takes 2 hours. It can't hurt!"

It can. We development managers learned it the hard way. We know how programmers think. We know how expensive switching their context is. If Sarah spends just two hours thinking of her old project, she loses a day of productive work on the new one. One day is 10% of a carefully planned iteration wasted if she spends 2 hours sidetracked.

In the wild nature of software development shops, however, it never takes 2 hours. 2 hours is the time Sarah is on the phone trying to clarify the problem. 2 hours is the time she is waiting for this phone call, reluctant to get into anything serious. 2 hours is the time Sarah is tweaking her development environment to build her old project. 2 hours is the time Sarah is spending to see if she can come up with a very restricted workaround. 2 hours is the time Sarah is on another phone call, explaining the potential workaround. Not enough time for real solution, no time spent on actually resolving the problem. 10 hours of unplanned and unproductive time is spread out over 3 days. 30% of iteration wasted.

At this point the planning goes down the toilet. The iteration is dead. The new project is slipping late. The rush around the old project yields little results either: with no time for a real solution the best bet there is a quick and dirty fix.

But the harm goes further. Sarah was eager to spend time on programming - she wasted it. She is robbed of her professional satisfaction, the good feeling of achieving the iteration goal and releasing project on time. On the next iteration planning session Sarah can't help thinking "Why kill time if we don't stick to the plan anyways?" The team gets the message: "We are NOT seriously doing iterative development. We are going ad-hoc".

Here is what I do:

* Tell the developer to stick to the original plan. Offer to protect her from switching her mind context. Remind her how cool it is to work single-mindedly on an important and interesting project. Ensure her that I'll handle the pressure. This is the good part; it is always well received :-)

* Tell the project manager that the iteration is carefully planned with a goal to release the new project by expected date. The plan leaves no space for switching the context daily. Our options are:
1) Put the issue to the back log and plan it for the next iteration.
2) Cancel the iteration, suspend the "new project", replan and work on the issue.
Let them make their call. This is the tough part; it is never well received. But I have to take a stand. I don't have the luxury not to do this.

The iterative development is an act of fine balance between adopting the continuous change and securing some stability for the developers to perform. We as development managers are responsible for keeping this fine balance. We do it for our team. We do it for our project. And we do it for our fellow project managers.

Cause we know something they don't always know: "How two hours can waste two weeks"

First published on SoftwareFrontier

It's great to have all the visitors here from Reddit and Joel on Software - thanks!. Joel brings up a great question about the other side of this: what about the sales guy who wants the new feature?

Posted by Dmitri Zimine at 02:22 PM | | | TrackBack

The Essence of Agile

What is the difference between an agile process and a non-agile process?

Both are concerned with delivering value quickly and with high quality. No one can dispute that those are "Good Things".

Both, in their own ways, have mechanisms for learning how to deliver more value. In iterative and incremental processes, this is done by checking the results against reality and adjusting the work to be done in the next increment. A non-agile process can do this just as easily as an agile process.

The real difference is in learning at the process level. A non-agile process doesn't change. An agile process starts with certain artifacts, meetings, steps, roles, and those all are up for discussion and change based on learning about the work. You can't freeze a definition of an agile process in reality. (It is, of course, possible to define an agile process starting point.)

Any other differences?

Posted by Mishkin Berteig at 10:52 AM | |

November 07, 2006

Scaling Agile Projects

More and more, organizations are applying agile methods to large projects or efforts that require more than a single team. There are three dimensions or concerns of coordination. It is critical that all three be addressed, but there are many ways of addressing them. Here I will simply list these three types of coordination and make some simple suggestions of how to implement them.

I have now had the opportunity to work with several clients where they are applying agile methods to projects with budgets that are greater than $10,000,000. All of them are using multiple teams to work on the same overall project/program. Out of this experience (along with some good reading along the way), I have come to understand that the following three types of coordination are the essential ones:

Value

In order to maintain the "early and frequent" delivery of value that agile methods propose, it is important that the work of the effort be coordinated to maximize early delivery of value. From this perspective, there are often many cooks in the kitchen. I have seen a "Product Owner Team", a "Customer Team", and a number of variations of this type. In order to do the coordination work effectively, it is still necessary to make sure of two things:

  1. Maintain a single Work Queue that prioritizes the work and from which all the teams select items.
  2. Have a single person in the "buck stops here" role who can make final binding decisions about work priority and content.

These two items have some implications for the organization.

First, the teams must be organized to be generalist: each team should be able to handle any item on the Work Queue. Not every team is going to be equal in abilities and this can be accomodated in a number of ways. My favorite so far comes from an excellent agile coach Dave West who suggested that teams bid on the items in the Work Queue at the start of each iteration. This should be done in a collaborative fashion so that it isn't just a simple low-bid-gets-the-work, but rather the teams learn from each other and have an opportunity to adjust their bids.

The second implication is that the customer or product team (Queue Master) must have the availability to support multiple teams in a timely fashion. Ideally, there are individuals on each team who can make judgement calls about features, functionality, constraints etc. on the work and provide quick answers to questions. This is not always easy since the people doing this often have a special area of expertise and it is difficult for them to work outside this area. Just as team members are asked to become generalizing specialists, so must the people who are responsible for determining value in a project.

Process

An agile process endeavors to provide a minimally structured way to do three things: deliver value early, then learn about what is high in value and deliver that more, and finally, learn how to deliver value more effectively.

That third activity, learning how to deliver value more effectively, is facilitated by the Process Facilitator. The Process Facilitator keeps a visible list of obstacles and works collaboratively with the Team and the Stakeholders to resolve obstacles on the list.

In a multi-team environment, there may be a single Process Facilitator working with each team. Like with the Work Queue, it is often necessary to have a single Record of Obstacles for the entire project.

Technique

People develop skill and knowledge in the use of their tools. Most types of work have a special vocabularly that only makes sense to other people also doing that work. For example, the field of computer programming has programming languages, integrated development environments, build tools, testing tools, algorithms, and a host of other techniques. The field of film-making has cameras, film, directorial techniques, lighting, story structure, it's own esoteric vocabulary, and other techniques. Likewise for construction, law, medicine, drama, education, etc. etc. etc.

In a large Agile Work project, teams need a way to coordinate their technique to produce integrated, consistent and compatible results. As well, individuals on the teams may discover or create new ways of doing things that would be valuable for the other teams to know about and use.

The most effective way of coordinating technique across teams is for strong members of each team to gather regularly to review the way work is being done. This "technical support group" can look at tools, reuse, automation, patterns, vocabularly and any other "how to" aspects of the work. It is critical that these people are actively involved in work being done on a delivery team so that efforts of the technical support group do not become academic or "ivory tower".

In certain environments, it may be useful to have this techincal support group become a team with a clear allocation of time apart from the regular delivery teams. In this case, this technical support team would have its own Work Queue that consisted of requests, ideas, concerns, and opportunities presented by the regular delivery teams.

I have seen all three aspects of coordination implemented in large multi-team projects. Some of the common challenges include:

  1. Generalist Teams.
    It is difficult enough to create cross-functional teams where people are willing to become generalizing specialists. While it is important to create generalist teams, most organizations should expect to set up non-ideal specialist teams (sometimes by line-of-business) and support their development into generalist teams.
  2. Technical Coordination.
    Often organizations have a design or technical review group composed of the "experts". These people are often isolated from the actual work being performed by the teams. It is difficult, yet critical, that these people actually be involved in day-to-day work on the teams.

Posted by Mishkin Berteig at 09:15 AM | |