Agile Advice - Working With Agile Methods (Scrum, XP, Lean)

I’ve worked as an agile coach on three infrastructure/maintenance projects in a row. One was a software/hardware upgrade, one was implementing agile for a defects/enhancements team, and my most recent was a data warehouse decommissioning project. In all cases, the interesting part for me was taking the basic principles of agile and applying them in a way that works when not doing new product development. Here are some lessons I’ve learned:

1. Figure out what is going to deliver value (usually cost savings). In the case of infrastructure projects, one is usually focused on cost savings. Find a way to tie your work items directly to cost savings. You need a good financial model to do this. Mary and Tom Poppendieck talk about this a little in their Lean Software Development book. In the decommissioning program, there was a very explicit dollar cost associated with disk space and cpu utilization. Every user/MB decommissioned saved a measurable amount of money. As well, it allowed us to easily prioritize our backlog.

2. Focus project/program organization more on Lean principles than agile. A good understanding of queuing theory will go a long way to helping with throughput. In a team doing defects/enhancements work, the small pieces lend themselves well to certain types of streaming through the team. Iterations are not necessary to chunck work. Instead, iterations become checkpoints solely for process reflection.

3. Technical infrastructure projects can benefit greatly from automation. Test automation including test generation can sometimes be possible. Automating parts of a regularly repeated process that is used for every work item can be extremely beneficial for increasing speed. In the case of the decommissioning effort where every database table needs to be considered separately and where they all go through the same process for decommisioning, there are many opportunities for automation. The project/program/team can invest in doing this automation to great benefit to NPV.

4. The basic axioms (We are Creators, Reality is Perceived, Change is Natural) and disciplines (Empower the Team, Amplify Learning, Eliminate Waste) still apply. Even though it is not “new” product development, the creativity of people is essential for problem solving, and finding ways to do the work faster. Stakeholders still need to have their perception of reality acknowledged, and the teams still has to do constant checking to make sure they are on track with that perception. And of course, things are always changing including priorities, our understanding of the work, resource availability etc. Having an empowered team makes short work of many obstacles, but that wouldn’t happen without an explicit acknowledgement that we have to constantly be learning and eliminating waste. Teams get better and better at these disciplines over time.

I would be very interested to hear other peoples experiences with infractructural/operational projects.

Detailed report on applying agile practices to an architectural project: http://www.architecturalpractices.com/articles_report_001.html. An interesting excerpt:

One of the things I like about Extreme Programming (XP) is that its advocates gladly share their experiences using it. This sharing contributes in many ways to its increasing value as a project management process. We can all review, comment, and learn from each other.

With this end in mind, I offer the following report on an architectural project completed recently using XPM practices. XPM is an acronym for Extreme Project Management, my version of a set of practices based on Extreme Programming. It shows one way XP practices can be adapted to a non-software project. I hope others using or considering similar practices will find this report useful for their own endeavors.

I’ve been working on developing a Agile Work Seminar to introduce teams to agile work. I’m using some Agile Work practices to develop it.

Iterative Delivery and Adaptive Planning

The seminar is going through drafts. Each draft will actually be delivered to a team. The first time through all the material was done at a small software consulting company five days ago. As a result of feedback from the people who participated, a revision will be made to the seminar… and then it will be delivered again (probably the next time will be in early September). This process allows me to refine the contents and presentation of the material.

Over time, I will be able to use Adaptive Planning to modify the contents and qualities of the seminar as circumstances change.

Test-Driven Work

I have set up criteria for the presentation in the form of an outline and learning objectives. The outline describes the major topics that must be directly covered such as the Agile Axioms or Corporate Culture. I have also set up “soft goals” such as that the seminar must include theory, history, practice and criticism of Agile Work. My first iteration met the outline tests, but did not meet the soft tests explicitly. The next version will.

Appropriate Metrics

This is an easy one: the success of this project will be its acceptance in the marketplace by having teams willing to pay the price for this seminar and then recommending it to others.

The Other Practices

Because this is essentially a one-man job, the other practices such as Self-Organizing Team and Maximize Communication are not as applicable.

A while ago I posted the story of a student film project using agile practices to create a documentary. One interesting observation made by the instructor is that trust among the group developed in an interesting fashion.

At first, the group self-organized by try to work in groups of three. However, when plans were made to get together (for example to film an interview), often, one of the three people would cancel. Probably, that person considered two people to be enough to do the work.

After noticing this pattern, the group decided to perform work in pairs. This made the commitment to working much stronger and eventually led to a more trusting work relationship.

I have also observed this pattern in other situations. Pair programming, pair writing, pair designing, pair problem-solving… all of these behaviors seem to arise naturally in a self-organizing team.

I recently had the pleasure of coaching a team through the use of agile methods to run an IT infrastructure upgrade project. As a coach, I was there to observe and guide, to inspect and adapt. The goal of the project was to upgrade a critical piece of software that was at the end of its support life, and at the same time upgrade another piece of software to a more recent minor version. As well, new hardware would be used to run the updated software. All of this is in the context of a mission-critical system used by a major US financial services corporation. Let’s call this project the “System Renewal” project or SR.

Prior to deciding to use agile methods, the SR project was defined using two non-agile requirements documents which listed in detail what the results of the project shall and shall-not be. One was the “business” requirements and one was the “system” requirements. For example:

SR-BR 0030
Description: The SR upgrade shall maintain the same or better level of performance.
Requirement Priority: Essential
Source/References: Joe Somebody
Comments: The SR version x.01 shall be upgraded to the x.02 as a part of this project.

I worked with some key people to review the project requirements before beginning. We decided that we would hold a workshop with the team to accomplish three things: introduce the team to agile methods, provide some initial team-building opportunities, and to introduce and launch the project with the team. The team decided to use two week iterations and to shoot for a single release after four iterations. Since this was much faster than the time estimated using the old waterfall methodology, there was a large margin of safety in case a second release was necessary.

During this workshop, the team re-organized the requirements into larger “user stories” and smaller tasks. This process of re-organization allowed the team members to collaborate and to get to know the nature of the project in more depth. There was a challenge here: the work to be done had nothing to do with end users. The team along with some guidance from myself and some other coaches, decided that the “user stories” would be very similar to the top level of a traditional work breakdown structure (WBS) and the tasks would be the second level. As much as possible, dependencies among the user stories were avoided so that they could be re-organized from iteration to iteration as circumstances required.

Once the project launch workshop was complete, the team immediately moved into the first iteration planning meeting. In this meeting, the team estimated the size of the tasks and selected the user stories to be completed in the first iteration. The basic idea was to accomplish one main goal: to upgrade the software on a small number of systems (the total number of systems to be upgraded was about 15). A secondary goal was to make some progress on automating a set of system tests to include in a regression suite. Prior to this project, all the tests were done manually. No new tests would be created as the functionality of the system would not be changed.

The first iteration went fairly well: from our burndown, it appeared that not much was accomplished in the first six or seven days of the ten day iteration. The last few days were extremely productive and the main goal of the iteration was accomplished. Some progress was made on automating tests, but that was not actually finished so manual tests were used to confirm the operation of the upgraded systems. During this first iteration I acted as the team’s coach by facilitating the daily status meetings, maintaining the burndown charts, and trying to resolve impediments to the team’s progress. At this point, impediments were mostly related to the physical and techinical environment: lack of phones, whiteboards, system access, etc.

The team had a half-day iteration reflection meeting that focused on questions about estimation and tracking of task status. The customer representative accepted the work that had been completed although an actual demonstration was not performed.

The second iteration was planned as the first. A few days in, the second challenge was encountered: the existing manual testing revealed that the new version of the software had modified some interfaces with other systems. As a result, the software did not work. The team temporarily halted its work in order to examine the problem and try to determine a solution. The team quickly got in touch with the vendor of the software and a simple fix was planned that would be deployed near the end of the second iteration. Some re-organization of the iteration’s user stories and tasks was done in order to accomodate this problem. This was done in collaboration with the customer representative.

At this point it should be noted that this was a first for the organization: testing early revealed a problem that normally would be discovered right at the very end of the project rather than right at the beginning. This allowed the team to re-organize their work to remain productive and so there was no change to the schedule. This is one of the main benefits of the combination of iterative delivery and test-driven work: the early discovery of problems and the ability to work around them without substantial schedule or scope impact.

The second iteration concluded successfully. The third and fourth iterations were relatively uneventful. And the team deployed the upgraded systems more than two months early. No new software was created. The key practices used either fully or partially included: self-steering team, iterative delivery, adaptive planning, test-driven work, and maximize communication (use of both co-located team and information radiators). The question of appropriate metrics did not come up except insofar as the team was substantially exempted from the corporate procedural compliance measurements, and was operating in an environment where time-to-market was considered the most important factor.

The Roots of Lean [PDF] is a fabulous history of how lean principles and practices originated in WWI shipyards in the United States, grew through WWII, were transferred to the Japanese and then lost in the United States.

So it turns out, in our house, our planning horizon is about one day. Therefore, our one-week iterations were consistently having problems: lots of dropped tasks, but still very busy. As well, the effort of trying to maintain a huge backlog revealed some environmental constraints: we spend a lot of time on yard work, for example.

As a result of these two discoveries, we have moved firmly to day-long iterations. Each day at the beginning of the day we take between 5 and 15 minutes to plan out our days TODO list. Melanie and I each have separate lists. We coordinate our lists throught the master household backlog. Melanie does very well in choosing a TODO list that she can accomplish in a single day, partly because she uses her daily planner to schedule things (estimation). I tend to have a lot of small things fall off the end of my list at the end of the day, mostly because I do not do any sort of estimation or scheduling. I may start.

In my previous entry about Agile Household Management, I mentioned the use of a corkboard. That is no longer being used for tasks. It is now being used for general notices and lists relating to larger tasks. For example, a list of items we need to purchase from Ikea, and a list of repairs we need to hire a plumber to do. My iteration (daily) TODO list is now a piece of paper at the side of my desk.

Briefly: I have a story to tell of two organizational cultures. In one, called Great Leader Financials, management tends to trust people in the company enough to give them great lattitude in how they get work done. In another, called Rigorous Process Financials, trust all around is low and is compensated for by meticulous processes and documentation. Both organizations are large with many thousands of employees.

At Great Leader Financials, our project team decided we needed to put in place a new server to experiment with some software tools to assist with the work we were doing. One of the team members had recently finished with another project and knew that the other project had a server they were no longer using. He simply contacted someone in management, let them know what was going on, and re-configured the server for our team’s use. Within minutes we had effectively transferred “ownership” of the server from one team to another. Several hours later, the server was set up with the new software and we were able to start trying it out.

Regrettably, in a similar situation at Rigorous Process Financials, our project team identified a need for a server to try out some interesting software tools for our project… and then ran into a huge bureaucratic brick wall. It turns out that at Rigorous Process, you can not just re-configure an existing server. One must follow the prescribed process. Unfortunately, the prescribed process requires getting a budget and project approval. And getting project approval involves going through a very lengthy project prioritization process. And small experimental infrastructure projects are always given a very very low priority. Trying “guerilla” tactics would have fared no better. Simply going out and purchasing a cheap computer (say $400 bucks) and brining it in would have only one effect: bringing down the immediate wrath of the IT Operations people as they detected a new server on the network. At Rigorous Process Financials, we do things the right way, or more likely, we don’t do them at all.

In January of 2005 I was invited to give a one-hour introduction to Scrum to a college media arts class. The class had one group project: to create a student documentary film on some topic. The instructor and the class used an adaptation of Scrum to manage the work of creating the documentary. Some key features from Agile Work included Self-Steering Team, Iterative Delivery, and Adaptive Planning. The following is one student’s reflections on the project. The student who wrote the following was, during my presentation to the class, acclaimed as the best person to be the Scrum Master.

“The whole documentary process has been an enlightening challenge.

First there was the process to find a consensus on a direction. Possibly the first real challenge as a group. As scrum master there were some great personal challenges. I am not a big fan of teamwork. I see it’s value but I have always worked independently and lived or died by my own efforts. There are huge issues of trust and success when you start with a group dynamic. The trust goes both ways. What if I let the group down because of my health or slower technical skills?? When you work independently you can set your own schedule. Will the pressure of letting down the group be enough to motivate the individuals?

SO the second big learning process for me was to be part of a team.
As scrum master I needed to keep myself focused and everyone on track without being a too pushy. NO one has any real obligation to me or anyone else on the team so setting a manageable pace, problem solving as an individual and as a team was another vital part of the process.
The whole concept of iterations kept things in manageable sections, that was a huge contributor to keeping a manageable goal in site when we were all feeling overwhelmed by the load of all of our courses work. AN interesting observation was that I think there were times when what we were doing in documentary was relatively minor but the fact that we were accountable for it everyday made some people feel overwhelmed. I certainly stand to be corrected but I think because we became such a good team that there was (and still is) a sense of the collective workload.

We all improved our technical skills, me particularly. I still have lots to learn but my confidence level is much higher.

There is a lot of respect shown for everyone’s work. We joke about things but people are mostly very respectful of the time and effort people put into their pieces of the puzzle. There doesn’t seem to be a lot of ego attached individually. We are each willing to recognize each other’s strength’s and weakness without rude or critical.

Overall this has been a positive process that is all about teamwork, individual strengths and respect. Garry [the instructor], you have been terrific about leading from behind! There were many times (and still will be) when we know we have a direction but don’t know how to get on the road exactly. You share your professional experience in such a way that we feel like we went searching for buried treasure and made the map ourselves.

I hope to finish the process and keep the momentum to an exciting finished product.“

Sharon

The project was indeed completed successfully! Other students in the class also provided very positive feedback on the use of Agile Work methods.

In a previous entry, I described some problems that we are having in using agile work practices in our household management. We have put up a cork board in our hallway in order to hold our tasks. It’s lovely! It also is really practical. As we complete the tasks, we are taking them down and throwing them away. As might be expected, this very physical closure on tasks is very satisfying. For myself, the visible tasks has greatly improved my awareness of what needs to be done. I find myself checking the board a few times a day at least. It remains to be seen if this will help us become more accurate in our selection of work for an iteration…

My wife and I have been doing weekly iterations to deal with our household management stuff. As we go along, we are definitely getting through the high-priority items on our household backlog. Along the way, we have encountered a couple of glitches.

1. We missed doing our weekly planning meeting one week. Basically, it was a really busy weekend with way too much stuff going on, both expected and unexpected. We simply failed to remember to do our planning meeting until it was too late to fit it in.

2. Most weeks we miss a number of items that we have selected to complete that week. It can be anywhere from 10% to 70% of the total number of tasks. Usually we get the “larger” tasks done and it is just smaller stuff that we miss.

3. I don’t have nearly the same visibility into the work as my wife does. I travel and am away from home four days a week. As a result, I don’t get to see the state of the house or participate in daily planning except for the three days that I am home.

So, we’ve discussed these things and decided that one thing that will help is to create an information radiator. We will be putting up our weekly selection of tasks on yellow sticky notes on a prominent wall in the hallway between our bedrooms and the rest of the house. The location is a compromise between visibility and privacy. We don’t want the tasks to be in a public portion of the house.

We hope that having the tasks up will allow me to be a little more conscious of the work that needs to be done. As well, it will be a frequent reminder of what is left to accomplish. I hope that this will be a fairly easy and valuable addition to our agile household management process.

Four weeks ago my wife, Melanie, reviewed a paper I was writing about Agile Work. When I talked to her about it, she asked me why we aren’t using these practices for managing our household.

(more…)