Product Backlog Items are ordered into a sequence in the Product Backlog in such a way that the Product Owner is able to maximize the return on investment (ROI) in the team. The very first PBI in the Product Backlog should be the one with the highest expected value considering the effort to build the PBI. There are many ways to calculate this expected value including Return on Investment (ROI), Net Income After Taxes (NIAT), Net Present Value (NPV), etc. The Scrum Team members should be free to ask why one PBI is prioritized higher than another, and the Product Owner should be able to give a reasonable answer. Since the entire Scrum Team is accountable for its work, it is in the best interest of all members of the team to use expected value, so that both the Scrum team and the customer will be committed to the work that is currently being worked on and the upcoming work in the future Sprints. If we don’t order the PBIs by expected value, then the Product Owner is likely to prioritize them based on dates, feelings, urgency, or other less valuable methods. These other prioritization methods will diminish the trust of the team in the Product Owner and may lead to morale problems.
The Product Owner brings Product Backlog Items to the Scrum Team to estimate their effort (cost). In order to create the right environment of safety and accountability, no Product Backlog Item is estimated by a single member of the Scrum Team, or even a subset of the team membership. By having all the members of the Scrum Team participate in the estimation work for every Product Backlog item, it becomes impossible to blame a single Team Member for a poor estimate. At a practical level, it is should be very rare that a single Product Backlog Item is fully implemented by a single Team Member. Therefore, estimates should consider the collective effort of the Scrum Team, and this can only be determined by having all the Team Members participate in the estimation work. If the team delegates estimation to a single person, or if one person dominates the estimation work, the other Team Members will not have ownership of the estimates and will be able to deny accountability. The pressure on the team from collective estimation encourages teamwork, cross-training and these behaviours in turn promote the development of a high-performance team.
The Planning Game [PDF] – printable reference.
Purpose: estimate the effort for User Stories (Product Backlog Items, Value Drivers)
Prerequisites: all items have a value estimate, each item is written on a separate note card, full team membership is known and available for planning, each team member has a set of planning game cards:
(Please feel free to contact us if you would like some sets of Planning Game cards. We will normally ship them to you at no cost!)
The Planning Game Process
- The team goes through all the items and chooses the one which has the lowest effort. Write the number “2” on this card (usually in the bottom right corner).
- The team looks at the item with the highest value.
- Each team member thinks about how much effort the team will expend to fully complete all the work for the item. Comparing this work to the work effort for the smallest item, each team member selects a card that represents this relative effort. For example, if you think that it requires ten times the effort, you would select the “20” card. It is not permissible to select two cards.
- Each team member places their selected card, face down, on the table. Once all team members have done this, turn the cards over.
- If all team members show the same value, then write the value on the item and go back to step three for the next item. (Or if there are no more items, then the process is complete.)
- The person with the highest and the lowest value cards both briefly explain why they voted the way they did. If there is a Product Owner present, this person can add any clarifications about the item.
- For any given item, if a person is highest or lowest more than once, then each explanation must include new information or reasoning.
- Once explanations are complete, the team members collect their cards and go back to step three.
– it is extremely important that the voting for an item continues until all team members unanimously vote the same way (this way team members and outside stakeholders cannot blame any individual for “wrong” estimates)
– in Scrum, it is normal for the Product Owner to be present during this process, but not to participate in the voting
– in OpenAgile, it is acceptable for people serving as Growth Facilitators for a team to participate in the voting
– voting should not include extensive discussion
– if more than one person has the lowest or highest vote, usually just one person shares their reason in order to help the process move quickly
– the first few items will often take 10 or 15 rounds of voting before the team arrives at a unanimous vote
– later on, items may take just one or two rounds of voting to arrive at a unanimous decision
– some teams, where trust levels are high, will discard with the use of physical cards and just briefly discuss votes
The planning game is used at the start of a project with the full list of user stories. In this case, it is reasonable to expect the team to average two minutes per user story, and an appropriate amount of time needs to be set aside to accommodate going through the whole list.
The Planning Game is also used any time that there is a change in the list of user stories: re-ordering, adding or removing user stories, or changes to a single user story. When such a change happens, the team can re-estimate any user story in the whole list. When starting a Cycle or Sprint or Iteration, all the user stories in the list should have up-to-date estimates so that estimation work is avoided in the Cycle planning meeting.
Finally, the team can decide to re-estimate any user stories at any time for any reason. However, it is important for team members to remember that estimation is non-value-added work and the time spent on it should be minimized.
NOTE: The Planning Game is described as Planning Poker on wikipedia. The version described there has some minor variations from this version.
A closely related method of Agile Estimation is the Bucket System.
Here’s a fun article on PMI.org. By omission, it gives some very bad advice about estimation. What is it missing? Asking the people who are going to do the work!!! Any estimation method or approach that fails to ask the actual human beings who are going to do the work about the effort required is going to be badly wrong. (Of course, even asking the people who are going to do the work is no guarantee of good estimates.)
The starting point of the linked article is a study that showed 90% of projects having cost overruns. To me, this just shows the futility of predicting the future, not anything about how we can (and should?) make better estimates.
The question of “expected velocity” and long-term planning has come up at more than one client. A recent client conversation got me thinking, however, questioning how to interpret velocity when estimating and plotting a roadmap based on a current backlog of features. Assume, for a moment, a backlog of story-pointed features, and 10 good iterations (consistent team, no odd occurrences that would affect velocity). Mathematically average velocity (well, a mean really) is a 50/50 proposition for any subsequent iteration. Some organizations don’t find this level of confidence acceptable. What velocity should be reported as expected for iteration/sprint planning and roadmap forecasting, and how should it be used?
Interpreting velocity, before anything else, requires some context. An agile organization that sees estimates as hypothetical might find this article is of less use. In fact, a good question is whether estimation is even a value-added activity. For this post assume an organization that sees strong value in estimation and planning.
The biggest piece of context is to know the organizational culture. This is important in two respects, and both of these cultural factors are important because they impact how Velocity is understood within the organization.
What is Failure?
First is the meaning of failure in the organization. Is failure to deliver what was committed to by the planned date considered a failure of the team, or is it simply a fact to be understood and accounted for in future planning? Even in Agile organizations, the former is often true and a hard habit to break. If not delivering to expectations is considered failure and has negative consequences, then that means that estimation is being treated not as estimation, but as prediction and contract. Velocity is therefore a commitment, and should therefore be used conservatively.
Consistency or Speed?
The second item to know is whether consistency and predictability of delivery is of a higher strategic value than the actual rate of delivery. This is often un-stated. Usually people want fast and consistent delivery. The truth is that you can get consistent, or fast software development, or a balance between the two. Lack of trust is usually a strong motivation to encourage consistency over speed, or a history of quality problems, etc. In this case, as well, Velocity is more of a boundary than an indicator.
Emotional Loading in Estimation (or why not Low-ball?)
If estimation is seen as binding, contractual, or limiting, then additional emotions get overloaded. Trust, promise, and betrayal are words used in such organizational cultures. Distrust is usually a strong factor, especially between silos (business vs. technology, company vs. project management vs. customer, etc.). So when people are asked to give estimates, even using agile-friendly mechanisms such as story points, there is usually a process of cementing that estimate into a part of an accountability model, so estimates start to get conservative. People are then accused of low-balling, others are accused of irrational expectations… we’ve all seen this. The language clearly becomes one of contention and blame. Even the term low-balling is often an outright pejorative term for estimating too conservatively.
This doesn’t happen only in agile environments, and project managers in traditional PMBOK frameworks have long factored risk into “contingency budgets”. Interestingly, however, if a Project Manager were to factor risk into the task estimates, they’d be “low-balling capacity,” yet if they were to factor it out and layer it on top of the project work, it’s “contingency budgeting” (At least in a few experiences I’ve had). Either way, someone’s adding a factor for uncertainty, based on the need to predict conservatively or liberally or somewhere in between.
That’s the point of the article: how can Agile projects use velocity to estimate as conservatively (or liberally) as is appropriate?
An average is a 50% chance to succeed (or fail)
Velocity is not a constant. It’s a set of instantaneous values on a curve, with instances being iterations. That means that it varies, and is therefore only meaningful statistically. So how do you reasonably use velocity statistically, and improve confidence? One way is to stop delivering against “average” velocity.
A lot of coaches use average velocity over the previous N iterations. This is not helpful for all sorts of reasons, if estimation is a commitment. By definition, average (well, actually a mean, but they’re close) is a 50/50 proposition. If you report the average team velocity (assuming it’s accurate), then about half the time the team will be under and about half the time the team will be over, statistically. So basically an average is a crap shoot, when taken in any given instance. It’s can only be good in the long run. For this to work, the long-haul has to include permission to fail and a lot of trust. Teams need to be able to go miss dates but will sometimes exceed dates and it should all wash out in the end. In organizations such as I’m describing, that trust isn’t there, so. Additionally, if the language of commitment is around meeting instantaneous iteration commitments (as opposed to delivering high-quality customer value as quickly as is sustain-ably possible) then you aren’t playing the long-game, you’re playing a very short-game.
Simulate Velocity, not work
In a PMI training course I took when I was at Sun Microsystems, we were nicely informed that two point estimates of tasks are a perfect way to fail half the time, per the above logic. One point estimates are just idiotic. Three point estimates were better. We simulated with a monte-carlo algorithm and found a curve and a distribution, and then determined a confidence level yadda yadda. Well, we’re trying to avoid wasting a lot of time estimating up-front, but one way to start representing velocity properly is to do the same kind of statistical modelling done in traditional product management, only simulate velocity, not work items.
In this approach, you take the last N iterations (say 10). Determine the maximum velocity (optimistic) and the minimum velocity (pessimistic), and then the mode (the velocity value that seems to occur most frequently). Then you do monte-carlo simulation so you get a statistical pattern. Now, you actually can determine an answer based on confidence. If you want to be right with an 80% confidence, you pick a velocity where 80% of the simulated runs were successful. (Note – there are a paucity of excel templates to do this math automatically, and often they are for sale. It would be nice to have a few functions with arbitrary distributions based on min-max-mode to help this along.)
It’s not perfect, and it’s a potentially huge amount of administrative overhead. Elsewhere I’ve referenced blogs that entirely oppose any estimation at all, but if you are gong to, then working statistically with simulation is the only way to take small sample numbers meaningful.
Commitment Velocity: Low-Ball as a policy.
Another approach, one perhaps controversial, but taught by some Scrum trainers is to pick the lowest historical delivered velocity. This is a commitment-based approach, on the assumption that building trust around consistent delivery is critical to building sound relationships where product owners and teams can safely state their needs and get things done with a minimum of contractual behaviour. By taking the minimum, you force a low-ball capacity, which means you can have high-confidence of success after a few iterations. You have, likely, after a while, some spare time on your hands. Teams can then choose to pull more work in (without adjusting their commitment velocity), work on “technical debt”, improve their skills, etc. A team could raise their commitment velocity in certain inflection points in the project. A new team member is added that provides a necessary skill not previously available, and after a few iterations the team is consistently hitting a higher number, but this is a careful process to ensure that they are committing, and if they don’t make their new number, it goes down to what they got accomplished.
Indemnify teams’ learning
An arguably healthier option, if you have built enough trust, is to simply indemnify a team from failing to meet the estimate. Since you’re doing mathematics on actuals to generate an expected future number, everyone can acknowledge that past behaviour is no guarantee of future behaviour, and simply use it for capacity planning. In this case, estimation is actually estimation, not commitment or contract. The team is expected to be ahead sometimes, and behind sometimes. The upside of this is that a lot of extra time isn’t spent playing with fictional numbers. Teams are spending their efforts on delivery as quickly-yet-sustain-ably as they can, and the organization treats them as trusted professionals in this. The temptation to assume you can predict the future is seen as folly, and the estimates are used to guide overall direction, not to make outward customer commitments.
Don’t be mindless
There may be other approaches, I’m sure. The agile community is certainly not short of people who love this topic and can talk for hours on “proper” estimation. The point of this post is merely to point out some options, and ask you to look at your organizational culture, team culture, customer culture, the meaning of terms like commitment, failure, success, consistency, speed, etc. As you understand the culture, balance consistency vs. speed, trust, and other factors to choose a method of estimation that meets your goals. Don’t do estimation based on your own, internal cultural assumptions, as you may have developed or been taught techniques that are useful when and where they were taught, but may no longer be so. Or maybe they weren’t so useful then either. Regardless, this because estimation cuts at the heart of the dialogue between producer and consumer, and establishes parameters for that discussion, it’s critical that you think your choice through.
[Christian also blogs at http://www.geekinasuit.com/]
There is an interesting article called “Software Estimates and the Parable of the Cave“. It starts out well. The cave parable is effective and clearly conveys the problem with estimating software work. However, there is a big section of the article called “Applying this Wisdom” which I think does a dis-service to everyone. Let’s look at this closely…
Kent Beck’s book “Extreme Programming Explained : Embrace Change” provides a good introduction to how software development can embrace the constant change that affects our world. Some of the practices he introduces are very software-specific. However, the overall basic message is sound and provides a foundational principle for all agile work. (By the way, the book is excellent.)
Change really does occur everywhere. Change is constant. A google search for “embrace change” or “change is constant” will both turn up an incredible variety of articles, papers, discussions, books and viewpoints that all affirm the constant nature of change and the need to embrace it.
Nevertheless, it is sometimes difficult to accomodate change when we also have a legitimate and deep desire to know what is coming next.
For many teams, the environment in which they work is constantly changing. This change can be caused by competition between organizations, scientific or technological advances, fads and cultural shifts, major events in people’s personal lives or even just a change of opinion with a stakeholder. Any change, even small change, can invalidate a planned course of action. However, goals (as distinct from plans) are more stable and often survive even major environmental changes. Therefore, rather than trying to plan the future, an agile team can focus on being able to respond to change while still reaching a goal.
Nevertheless, a team needs some sense of what it will do in the near future. A team can work with a “horizon of predictability”. This is the distance into the future which a team can be reasonably certain that plans will be stable. Depending on the environment, this may be as little as a few minutes, or as long as a month. It is rarely longer. The horizon of predictability is not a precise demarcation, rather, expect change with a probability based on the horizon of predictability. Then, plan to respond to change. Be detached from the concrete details of a plan, particularly if they occur outside the horizon of predictability.
Responding to change requires a major mental shift for many people that is difficult and takes time and environmental support. People are often penalized socially or formally for being flexible or adaptable. This quality can appear to be “wishy-washy”, uncertain, indecisive, uncommitted or even rebellious.
The terms “agility” or “agile work” refer to this principle of embracing constant change since it is the most visible of the principles. However, the ability to respond to change relies on the establishment of agile work disciplines and practices.