September 06, 2007
Four Methods of Perfecting Agile
Most organizations start doing agile (or Scrum or lean or ...) imperfectly. Someone introduces a few practices or a manager gets a team some training, or a person starts using agile terminology. And things might improve, particularly with the use of iterations. One of the core ideas of agile methods is to have frequent delivery of valuable results. In fact, this core idea can be used to drive the improvement of an agile process. How? Here are four methods of perfecting agile by expanding the definition of done.
Perfecting Agile
Let's suppose you already understand the benefits of agile. With these benefits in mind, you would like to improve the organization's ability to deliver working, valuable results at the end of every iteration so that you can get better at realizing those benefits. The primary way to do this is by expanding the definition of done. You can imagine this like so:
On a regular basis, the team/organization find ways to bring work done in either the preparation stage or the close stage into every iteration of the agile portion of the project. By moving the work from these "bookend" stages into the iterations, you reduce the amount of time spent in those stages and simultaneously create a more complete delivery every iteration. The "definition of done" is now expanded to contain the results or value delivered by the work that was taken out of the startup and shutdown stages of the project. By expanding the definition of done, each iteration delivers a more "complete" increment of value, and there is less work done before or after iterations in order to plan or deliver. This gradual process allows the team to get better at doing agile.
There are four methods for transferring work from the start and end of a project into the iterations of a project.
Expand the Agile Team's Skill Set
In some ways, this is the simplest and most common approach to expanding the definition of done in the agile portion of a project. By training, coaching, mentoring, re-assigning or hiring, a team's capacity to do work is expanded and used to expand the definition of done. As a simple example, a software developer might learn to use an automated unit testing framework and therefore expand the definition of done to include some amount of unit test coverage of delivered code. In general, training, coaching and mentoring existing team members should be preferred over adding people to the team since the addition disrupts the team's development and can increase communication overhead among team members.
Expand the Agile Team's Authority
Sometimes, a team is not able to do part of the preparation work or close up work because they are not authorized to do so. This may be a policy, a unspoken assumption or a bureaucratic procedure. By giving the team (or some person on the team) the authority to do this work, the team can find ways to do it every iteration instead of having to work through another non-team individual. Again, a simple example here is a situation where a technical person is given permission to talk directly to an end user in order to reduce the need for up-front requirements gathering and analysis and reduce the need for end-of-project user acceptance testing. The obvious challenge to do this is the question of trust (or lack thereof).
Automate an Existing Manual Process
Automation is often given far less than its due consideration. This is primarily be cause automating a process is an investment of work in and of itself. Fortunately, it is often easy to measure the ROI or savings involved with automation. In many agile environments, heavy automation is critical and a huge enabler for very short iterations. Automated testing, automated translation, automated build processes, are all common areas of improvement. Agile teams should always be looking for opportunities to automate their own work. In this way, the automation work itself is transformed from a separate project to a responsibility of the team.
Remove Wasteful Processes
There are some parts of the project preparation work and the project close up work that are pure waste! There is no independent value to these activities, nor is there indirect value to them. An excellent example of this sort of thing is an approval process that _always_ grants approval ("rubber stamping"). One insurance organization I worked with as they were converting to an agile approach discovered that their "second stage" approvals always allowed proposed projects to proceed. Since they often incurred a 4-6 week delay for this approval process, it became obvious that they should "get rid of it". Now, what they actually did is made it so that it became a parallel review process rather than a gated approval process; this was so that the true purpose of the activity could still be met: to help stakeholders understand the projects that were being worked upon. Here, there is no need to take this approval process and somehow work it into every iteration. An agile approach tends to increase the visibility of the work anyway, so it may be discovered later on that the review process can also be done away with.
Agile is often implemented in a limited fashion when it is first adopted by most teams and organizations. The four methods of expanding the definition of done can help a team or organization get better at doing agile and therefore reap more of the benefits of agile. These methods are simple: expand the agile team's capacity, their authority, have them automate manual processes and remove wasteful activities from the process.
Posted by Mishkin Berteig at 03:32 AM | |
August 16, 2007
Agile 2007 Conference Notes - Mary Poppendieck on Leadership
Okay, I'm only here for one day, and the first part of the day was me giving a workshop and then the next part was a presentation on Ruby and Selenium which was interesting (but unfortunately nothing to write home about) and then...
Mary Poppendieck spoke. Here are my notes.
The Role of Leadership
Notes from a talk by Mary Poppendieck with some commentary by Mishkin Berteig.
History:
Taylor: "The Principles of Scientific Management"
- Assumptions: workers do as little as possible, they don't care about quality, and they are not smart enough to know the best way to do the work.
-> experts therefore define the best way to do a job and you pay your workers extra to follow the expert's instructions
-> results in happy workers and profits for the company
!!!
Note: this is almost exactly the opposite of the Agile Axioms
Charles R. Allen: "Industrial Training"
- method: Preparation -> Presentation -> Application -> Testing
Here's another example of the Learning Circle.
- on the job training best
- 1917 this approach tested w/ shipbuilding training
- - train the supervisors to train workers
- - 88000 people trained
- wrote a book: "The Instructor, The Man and The Job"
- "If the learner hasn't learned, the teacher hasn't taught."
WWII: "Training Within Industry" (TWI)
- inexperienced workforce to build weapons and munitions
- train first line supervisors
- - job instruction: how to train
- - job methods: how to do the work
- - job relations: how to work with workers
- abandoned after the war
BUT
-> exported to Japan to help them rebuild
TWI Premises: Good Supervisor
- knows the work
- knows responsibilities
- skill in instruction
- skill in process/work improvement
- skill in leadership
Toyota Production System (TPS)
- in 1950's Toyota looked at Ford and GM and saw that they were far more productive than Toyota
- Taiichi Ohno (1912-1990) founded TPS
- - Just in Time Flow (eliminate waste)
- - Stop-the-Line culture (mistake-proof the process)
- - - originally made LOOMS
- - - 1st self-stopping automated looms that stopped with defects
- - Relentless Improvement
- - Books: "Taiichi Ohno's Workplace Management
about "Standard Work":- Standard Work
- - document the way work is done
- right now
- - often used by experts to create an ideal process - this is a
- big mistake
- - standards evolve
Deming: 1980 "System of Profound Knowledge"
- Appreciation of the System
- - never sub-optimize by focusing on one part of the system
- - look at everything: suppliers, producers, customers
- Knowledge of Variation
- - most variation is inherent in the system therefore don't blame workers for systemic problems
- - provide leadership in changing the system
- Theory of Knowledge
- - Scientific Method (PDCA)
- - Use data
- Psychology
- - don't use #'s for people
- - Skill, Pride, Expertise, Confidence, and Cooperation motivate people
Respect People
- move responsibility and decision-making to the lowest possible level
-> when workers are annoyed by their job... do they:
- - complain (stone cutters)
- - ignore it (making a living)
- - fix it (cathedral builders)
Leadership
(adapted from "The Toyota Way
Top Down and General Management Expertise:
-> bureaucratic: "follow the rules"
Top Down and In-Depth Understanding of Work:
-> task manager: "here is what to do and how to do it - now do it!"
Bottom Up and General Management Expertise:
-> group facilitator: "you're empowered"
Bottom Up and In-Depth Understanding of Work:
-> builder of learning organizations: "here's our purpose now let's do it and learn along the way"
- this last type of manager is where most of Toyota's managers fall
John Shook (www.lean.org)
Three Models of Leadership from the Lean Management and the Role of Lean Leadership Webinar:
- Old "Dictator": do it my way
- 1980's Empowerment: do it your way
- Lean: follow me and let's figure it out together
@ Toyota a leader:
1. acts as a teacher who knows how to do the job and how to teach it
2. gets each person to take the initiative to solve problems and improve his or her job
3. ensure that each person's job is aligned to provide business and customer value
Brilliant Products
- require deep understanding of both business and technical domains
- who decides?
- priority by committee: no responsibility
- marketing by checklist: no innovation
- behind every great product is a single person
- - great customer empathy
- - deep technical insight
- - distinguish between essential and incidental
- Chief Engineer at Toyota
- Product Champion at 3M
Leadership Roles for Products:
- Champion:
- - marketing leader
- - technical leader
- Functional Leader
- - job instruction
- - methods improvement
- - job alignment
- process leader (temporary): coaching
the process facilitator?
- project leader (temporary): funding/tracking
Case Study: Zara (Fashion Clothing Stores)
- from design to store in 2 wks
- twice weekly orders
- - deliver globally in 2 days
- - on hangers, priced
- - shipping prices are not optimized!!!
- manufactured in small lots
- - coops in Spain
- - West European labor rates
- - labor costs are not optimized!!!
In order for Organizations to perform brilliantly, there are 2 pre-requisites:
- everyone in management agrees what they want
- everyone in management agrees on cause and effect
-> does everyone on the Management team speak the same language?
Mary then went on to discuss the topics of Cost Cutting, Conformance to Plan, Standards, Utilization, Accountability, and Measurement and contrasted a non-lean with a lean approach or attitude. Most of these ideas can be found in essays on her web site www.poppendieck.com or in her books:
Lean Software Development: An Agile Toolkit for Software Development Managers
and
Implementing Lean Software Development: From Concept to Cash (The Addison-Wesley Signature Series)
Both of which are excellent.
Here is her pdf of the presentation materials.
Posted by Mishkin Berteig at 02:01 AM | |
March 05, 2007
A Better Iteration Structure
In my coaching work, I have often been asked a question about the planning process for iterations, that until just a few days ago, I would actually brush off!!! I didn't even realize I was doing this, it is only in retrospect that I see this. This question is simple: "how does a team plan for the improvement efforts that come out of the retrospective when they are supposed to be working at maximum velocity when implementing tasks directly related to the items in the work queue?" Or, more simply, "we don't have time for process improvements."
The source of the problem shows up in the normal* structure of an iteration. This structure is as follows:
Step 1: set a goal by selecting work from the top of the work queue.
Step 2: plan the execution of that work by doing a task breakdown and task-level estimation.
Step 3: do the work according to the tasks (this is the bulk of the time in the iteration).
Step 4: do a demo to review the work - use this demo to adjust the work queue if necessary.
Step 5: do a retrospective to review the "how" and come up with action items to make the work more effective next iteration.
Since iterations follow one after another, this structure when rolled up can be re-written in the following manner:
Action (step 3)
Reflection, Learning (step 4)
Reflection, Learning, Planning (step 5)
Planning (step 1 and 2)
Now if we look at the purpose of the steps in comparison to the type of learning activity taking place, we see that there is a big disconnect between the planning that happens in step 5 (the retrospective) and the planning that happens in steps 1 and 2 at the start of the next iteration. Not only that, but reflection and learning are separated into "what" and "how" so are done twice for each iteration.
Frankly, this causes a lot of confusion: I see it in my coaching when teams try to accommodate the two types of planning, and I see it in my training when I teach the structure of the iteration.
So what is to be done? Simplify the structure of the iteration. Instead of thinking of the iteration as a linear block of time that starts with step 1 and ends with step 5, think of it as a cycle that is dominated by Action, and punctuated regularly by Reflection, Learning and Planning. Now, instead of four separate meetings, there is a single meeting every iteration that combines the wrap-up of the previous iteration with the start of the next. Let's call this the Iteration Transition Meeting or the RLP Meeting or the Adapt Meeting... (suggestions welcome!)
The agenda for this meeting is very simple and contains the essential elements that exist in the "old" style. It starts with Reflection where the facts of the Action are examined: what did we build? how did we build it? how did we feel about what we built? etc. This moves fairly quickly to the Learning component where we try to understand what the facts are telling us: what went well? what needs improvement? what insights can we glean? what new questions are open to us? and of course, what did we learn? Now we are prepared to treat Planning holistically, with no "hidden" activities. We examine our Work Queue, re-arrange it, put new items on it, etc. and do our normal task breakdown and estimation. The difference this time is that both direct business tasks and process improvement tasks are included in the same planning activity. This makes task level (commitment) velocity more truthful!
I will admit that this is not something that I have tried yet. I have seen it done accidentally when people start to ignore the action items from the retrospective in the old iteration structure and instead include them in the task planning. When it happened, I thought it was "messy" because it broke the separation between "what" and "how" oriented activities. Now I think that this might be the best way to do things.
Anyone out there willing to give this a try? If so, I would love to spend some time with you working on it - helping, experimenting, and learning from it.
I should mention a couple important points that didn't fit in the above narrative. First, this insight happened as a result of discussions over the past week with my father, Garry Berteig who is using agile methods and the learning circle in his classroom environment. Second, Garry has been doing what I just described in his classroom environment; the main difference is that Garry provides a great deal of the Planning to his students by way of the progression of assignments/activities.
One other cool thing: I'm in Frankfurt as I write this on my way to Chennai to deliver a three-day Agile Project Management / ScrumMaster Certification course. I haven't done international travel since I went to Beijing last summer, and this is the first time I am going outside of North America for work! Fun!
* Normal here means Scrum and XP for the most part. Lean doesn't have this structure and as for other agile methods, I don't know them well enough to say if this is normal or not.
Posted by Mishkin Berteig at 04:16 AM | |
January 02, 2007
Top Ten Most Popular Entries from 2006
If you are new to Agile Advice, these are not just some of the most popular articles, they are also some of the best! Take a look around; you will find ideas to inspire you, challenge your thinking and maybe that one little thing that will make the difference in applying agile methods in your situation.
1. How Two Hours Can Waste Two Weeks - 25,617 unique views
This little hypothetical story by Dmitri Zimine was very popular on Reddit, and Joel on Software ranted a bit about it.
2. The Case for Context Switching - 2,936 unique views
My rebuttal to Joel's rant. Goes well with Dmitri's article. Emphasizes the idea of building trust in an organization.
3. The Qualities of an Ideal Test - 1,579 unique views
Six qualities that will help make your tests as valuable as can be. Similar to the ACID properties of databases or the INVEST properties of user stories.
4. The Pros and Cons of Short Iterations - 1,555 unique views
A few words that will help you decide how long to make your iteration length. This is an important decision, and most teams and organizations don't know the factors involved.
5. Five Signs of Trouble in an Iteration - 1,489 unique views
A good howto on using burndown charts to discover problems in an iteration.
6. Seventeen Tips for Iteration Planning - 1,427 unique views
A good list of hints and tips that can make the difference between struggling with iteration planning and having it go smoothly and effectively. This is a key part of the Agile Work process, so getting good at it is a high priority for any team new to Agile Work.
7. Change is Natural - "Embrace Change" - 1,397 unique views
A short riff on the universality of change. Also introduces the idea of the "horizon of predictability".
8. The Art of Obstacle Removal - 1,287 unique views
This is a good reference article on types of obstacles and methods for removing them... a critical practice for Process Facilitators.
9. The Seven Core Practices of Agile Work - 1,285 unique views
Agile Work is really quite simple. This is a concise list of the practices that make up this effective methodology.
10. Interview with Alistair Cockburn - Agile and House Renovations - 902 unique views
Applying agile methods to home and garden renovations! Learn a bit about how this luminary of the agile world has taken agile practices out of the software world and into the hardware world.
Posted by Mishkin Berteig at 06:32 PM | |
May 17, 2005
The Qualities of an Ideal Test
These six qualities of tests describe how to make a test as effective and as useful as possible. The qualities are similar in style to the INVEST qualities of user stories - but they don't form a nice acronym.
Decisive: The test contains all the information required to automatically determine success or failure. Manual inspection of test results is not necessary to determine success or failure. The test is expressed in a way that produces a pass/fail answer rather than a numerical or qualitative result. Decisive tests are often expressed as assertions.
Valid: The test produces a result that matches the intention for the work artifact under test. Test failure indicates failure in the artifact under test and test success indicates correct operation or configuration of the artifact under test. Simply put: the result of a test should be true.
Complete: The test contains all the information it needs to run correctly with a given test harness and work artifact under test. The test performs all activities and provides all the data necessary. The test does not require any input outside of itself in order to run.
Repeatable: The test always gives the same results if the test harness and the artifact under test are the same. The test is created in such a way that the result is deterministic. Even if the system under test is not deterministic, the test should be created to account for that (possibly with statistical analysis) and produce a deterministic result.
Isolated: The test result is not affected by other tests run before it nor does a test affect the results of tests run after it. The test and the test harness work together to clean up after every test is run. A collection of tests can be run in any order and always produce the same results. Any test that depends upon the results or side-effects of a previous test is not isolated.
Automated: The test requires only a start signal in order to run to completion in a finite amount of time. No manual intervention is required after the test is started. Tests that are automated can be put together into a test suite and run one after another without human intervention. Automation of tests requires the creation of a test harness system.
Update 20060106
I've added a little bit more detail to the above qualities. I also wanted to say a few words about my experience with testing:
I first started doing test-driven development almost seven years ago after reading the book Refactoring by Martin Fowler. I picked it up in Windsor Ontario while I was driving to San Francisco for my first contract position as an independent. I had to stay the night there due to the immigration office being closed so I read a lot of the book in my motel room that night. By time I got to California four days later, I was totally convinced that I should be using JUnit and refactoring for everything I did. Fortunately my project was ammenable to this approach. Over the next four months I built a Java JMS layer on top of Tibco Rendezvous. In the process I discovered methods for doing unit tests for asynchronous multi-threaded distributed functionality. And my tests satisfied all the above qualities. I delivered code with 100% test coverage and zero defects discovered until more than two years later when a very rare and obscure deadlock issue was found. Suffice it to say, I was convinced from a quality perspective.
But there is more. To build tests that follow all the above qualities, you need code that is testable. I have come to believe that testability is the most important architectural attribute of code for most software. The implications of having testable code include code that is easily changed, that is verifiable, and that is easy to understand. Refactoring plays a big role here too.
For getting a basic but very practical sense of test driven development, I strongly recomment Test Driven Development: By Example 
I have also written a couple of articles recently about quality:
Quality is Not Negotiable in which I talk about how to handle defects - by always treating them as top priority work items.
and
Technical Debt in which I expand on this common analogy between lack of testing and debt to show that technical debt is even worse than financial debt.
Posted by Mishkin Berteig at 06:48 PM | |
May 12, 2005
Appropriate Metrics - Continued
Lean Metrics... are Lean
In the book Lean Software Development by Mary and Tom Poppendieck, there are a small number of references to metrics: process cycle time, process cycle efficiency, business models. Lean practices focus very much on diagnostic metrics that are used to help people find and eliminate waste and improve value and quality. The other aspect of metrics is to measure up for purposes of motivation and performance measurement.
Conclusions for Scrum and Agile in General
Don't prescribe specific metrics!
Agile work is about an empirical process where a team responds to change, learns, and self-regulates. An agile team has the power to choose their own metrics for their own self-inspection. For upper management, the single economic driver that is part of the "Good to Great" process should be sufficient.
See also: Appropriate Metrics and A Metric Leading to Success.
Posted by Mishkin Berteig at 07:14 PM | |
April 27, 2005
Eliminate Waste
Waste is the result of activities or environmental conditions that prevent a team from reaching its goal. The opposite of waste is something that adds value (more, faster or higher quality) to the desired result.
The whole notion of eliminating waste comes from lean manufacturing. More recently, Mary and Tom Poppendieck applied this idea to software in their book "Lean Software Development: An Agile Toolkit for Software Development Managers". In this (excellent) book, the authors list the wastes of manufacturing and the wastes of software. Here I have summarized and generalized these types of wastes so that they apply in any situation:
| The Seven Wastes |
| 1. waiting caused by delays, unreadiness, or simple procrastination |
| 2. partially done work or inventory caused by sub-optimal workflow |
| 3. extra processing or processes caused by poor organization or bureaucracy |
| 4. defects and rework caused by insufficient skill, tools, inspection or filtering |
| 5. movement of people or work caused by physical separation |
| 6. overproduction or extra features caused by working towards speculative goals |
| 7. task switching caused by multiple commitments |
As wastes are eliminated or reduced, a team will function faster and with higher quality. However, not all waste can be eliminated. Sometimes waste is legislated, sometimes waste is an unavoidable byproduct of work, sometimes mistakes are made, and sometimes it takes a great deal of effort to eliminate a waste.
In order to eliminate waste, first waste has to be detected and identified, then the underlying causes of the waste have to be identified, and finally changes to the work environment need to be made to both eliminate the cause of the waste and the waste itself. Many agile work practices help with this process.
Value stream mapping is one particular tool that can be used by a team or organization to identify wasteful activities. The team describes the amount of time that work takes to go through each activity in their overall work process. Next, the team determines if each activity adds value or does not add value to the end goal. All activities are subject to speed improvements, and activities that do not add value are subject to elimination.
In order to determine the causes of waste, special attention should be paid to incentives and motivations. Wasteful behavior often exists because there is some incentive for people to do it. Sometimes these incentives are explicit, but sometimes they are the side-effects of other things going on in the team's environment. Changing the incentives can be an effective way of reducing waste.
By eliminating waste, the team will find it has reduced frustrations, and enabled greater productivity and creativity. The team will also increase its speed and delivery of value, and at the same time reduce defects.
Posted by Mishkin Berteig at 07:40 PM | |
April 13, 2005
Backlogs in an Organization - Levels of Queues
Queues of work form at three types of levels in an agile organization.
At the largest level is the project portfolio. The queue for this level contains all the projects that are not yet being actively worked upon by project teams.
At the intermediate level is the backlog of project functionality. The queue for this level contains packages of business function or infrastructure components necessary to implement business function. These packages are selected by a project team to fit into a single iteration.
The packages in turn are also elements in a queue. This smallest level of queue contains individual tasks required to implement all the business function and infrastructure that make up a selected package of work. The team members select tasks off this queue based on priority and dependencies.
In many agile methods, the queue management approach is fairly explicit at the intermediate and small levels. However, very little is said about the largest level. Some organizations have solved this by limiting the size of projects:
To be successful, high-tech CIOs recommend biting off projects in small chunks.... Gregoire notes that Dell is growing so fast that at the end of an 18-month project, the company would be significantly different from when it began. "A project has to take less than six months [to complete]. That's the only way we can make sure [it stays] with the business," he says. (http://www.cio.com/archive/120198/tech.html)
Posted by Mishkin Berteig at 06:19 PM | |
Backlogs in an Organization - Levels of Queues
Queues of work form at three types of levels in an agile organization.
At the largest level is the project portfolio. The queue for this level contains all the projects that are not yet being actively worked upon by project teams.
At the intermediate level is the backlog of project functionality. The queue for this level contains packages of business function or infrastructure components necessary to implement business function. These packages are selected by a project team to fit into a single iteration.
The packages in turn are also elements in a queue. This smallest level of queue contains individual tasks required to implement all the business function and infrastructure that make up a selected package of work. The team members select tasks off this queue based on priority and dependencies.
In many agile methods, the queue management approach is fairly explicit at the intermediate and small levels. However, very little is said about the largest level. Some organizations have solved this by limiting the size of projects:
To be successful, high-tech CIOs recommend biting off projects in small chunks.... Gregoire notes that Dell is growing so fast that at the end of an 18-month project, the company would be significantly different from when it began. "A project has to take less than six months [to complete]. That's the only way we can make sure [it stays] with the business," he says. (http://www.cio.com/archive/120198/tech.html)
Posted by Mishkin Berteig at 06:19 PM | |