These are the photos from my talk in London Ontario on April 27th at the PMI SWOC Symposium conference.
In reality, Kanban isn’t actually saving Agile nor is it intended to, nor is any thoughtful and responsible Kanban practitioner motivated by this agenda. What I’m really trying to convey is how human thinking about the business of professional services (including software development) has evolved since “Agile” as many of us know it was conceived around 20 or so years ago. The manifesto is the collective statement of a group of software development thought leaders that captured some of their ideas at the time about how the software industry needed to improve. Essentially, it was about the iterative and incremental delivery of high-quality software products. For 2001, this was pretty heady stuff. You could even say that it spawned a movement.
Since the publication of the manifesto in 2001, a lot of other people have had a lot of other good ideas about how the business of delivering professional services can improve. This has been well documented in well known sources too numerous to mention for the scope of this article.
Substantial contributions to the discourse have been generated by and through the LeanKanban community. The aim of Kanban is to foster environments in which knowledge workers can thrive and create innovative, valuable and viable solutions for improving the world. Kanban has three agendas: survivability (primarily but not exclusively for the business executives), service-orientation (primarily but not exclusively for managers) and sustainability (primarily but not exclusively for knowledge workers). Kanban provides pragmatic, actionable, evidence-based guidance for improving along these three agendas.
Evolutionary Theory is one of the key conceptual underpinnings of the Kanban Method, most notably the dynamic of punctuated equilibrium. Evolution is natural, perpetual and fundamental to life. Long periods of equilibrium are punctuated by relatively short periods of “transformation”—apparent total and irreversible change. An extinction event is a kind of punctuation, so too is the rapid explosion of new forms. Evolutionary theory is not only a scientifically proven body of knowledge for understanding the nature of life. It can be also applied to the way we think about ideas, methods and movements.
For example, science has more or less established that the extinction of the dinosaurs, triggered by a meteor impact and subsequent dramatic atmospheric and climate change, was in fact a key punctuation point in the evolution of birds. In other words, dinosaurs didn’t become extinct, rather they evolved into birds. That is, something along the lines of the small dinosaurs with large feathers hanging around after Armageddon learned to fly over generations in order to escape predators, find food and raise their young. Dinosaurs evolved into birds. Birds saved the dinosaurs.
There has been a lot of social media chatter and buzz lately about how Agile is dead. It is a movement that has run its course, or so the narrative goes. After all, 20 years is more or less the established pattern for the rise and fall of management fads. But too much emphasis on the rise and fall of fads can blind us to larger, broader (deeper) over-arching trends.
The agile movement historically has been about high-performing teams. More recently, market demand has lead to the profusion of “scaling” approaches and frameworks. Scaling emerged out of the reality of systemic interdependence in which most Agile teams find themselves. Most agile teams are responsible for aspects of workflows—stages of value creation—as contributors to the delivery of a service or multiple services. Agile teams capable of independently taking requests directly from and delivering directly to customers are extremely rare. For the rest, classical Agile or Scrum is not enough. The feathers just aren’t big enough. Agile teams attempting to function independently (pure Scrum) in an interdependent environment are vulnerable to the antibodies of the system, especially when such interdependencies are merely denounced as impediments to agility.
Some organizations find themselves in a state of evolutionary punctuation (the proverbial sky is falling) that can trigger rapid adaptations and the emergence of local conditions in which independent service delivery teams can thrive. Most large, established organizations seem to be more or less in a state of equilibrium. Whether real or imagined, this is what change agents have to work with. However, more often than not, the typical Agile change agent seems adamant that the sky is always falling and that everyone accepting that the sky is falling is the first step to real and meaningful change. This is not an attitude held by Agile change agents alone. This is a standard feature of traditional 20th Century change management methods, the key selling point for change management consulting.
Naturally, most self-identifying “Agilists” see themselves as change agents. Many of them find themselves in the position of change management consultants. But the motivation for change can quickly become misaligned: Change needs to happen in order for Agile to work. If you are passionate about Agile, you will seek to bring about the environmental changes that will allow for Agile to thrive. We don’t need to follow this path too far until Agile becomes an end in itself. It is understandable then that for some, Agile appears to be a dead end, or just dead.
But if there is a larger, over-arching historical process playing out, what might that be? Perhaps it has something to do with the evolution of human organization. Perhaps we are living in a period of punctuation.
For my working definition of Kanban, please refer to my previous article 14 Things Every Agilist Should Know About Kanban (this contains links to the Kanban body of knowledge, including Essential Kanban Condensed by David J. Anderson and Andy Carmichael).
For my working definition of Agile, please refer to The Manifesto for Agile Software Development.
Many organizations are attempting to use Agile methods. Banks, telecom companies, government agencies, and all manner of mid-size and small organizations. Most of these attempts are limited in that they think of Agile as a solution instead of as a culture. In this video, I explore some of the conditions for creating Real Agility.
This is the first video in a series of eleven that is oriented towards what managers need to know to create Real Agility in their organizations. The final two videos in the series are going to be content exclusively available to subscribers to our Real Agility Newsletter. Those final two videos are about “Dealing with Crisis” and “The Knowing-Doing Gap”. (Our newsletter also includes other great content including interviews – we are featuring an interview with Mary and Tom Poppendieck in just a few weeks!)
Great technique described by Shahin Sheidaei on his blog: Challenge, Estimate or Override (CEO) Game for Effective Estimations. It is much quicker than the Planning Game, and probably a bit slower than the Bucket System.
The team decides on how much work it will do in a Sprint. No one should bring pressure on the team to over-commit. This simply builds resentment, distrust and encourages low-quality work. That said, of course teams can be inspired by challenging overall project or product goals. A stretch goal for a Sprint is just a way to 100% guarantee failure. Even the team should not set its own stretch goals.
There are a few interesting principles that apply here. For example, the Agile Manifesto mentions sustainability:
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
The Agile Manifesto also points out the importance of trust:
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
Stretch goals are incompatible with both of these principles from the Agile Manifesto.
There are two types of stretch goals. The first type are those assigned by outsiders to the team. The second type are those which a team sets for itself. Both types are bad.
The worst extreme of this type of stretch goal is also the most common! This is the fixed-scope-fixed-date project deadline. In this type of stretch goal, the project team, doing Scrum or not, is forced to work backwards from the deadline to figure out how to get the work done. If the team can’t figure this out, managers often say things like “re-estimate” or “just get it done.” (Note: another thing that managers do in this situation is even worse: adding people to the project! Check out “The Mythical Man-Month” by F. Brooks for a great analysis of this problem.)
My anecdotal experience with this sort of thing is simple: quality suffers or sustainability suffers. I once worked with three other people on a mission critical project to help two banks with their merger. There was a regulatory deadline for completing the integration of the two existing systems for things like trading, etc. Fixed-scope-fixed-date. Coffee and sleepless nights were our solution since we tried not to sacrifice quality. We actually ended up working in my home for the last few 24-hour stretches so that we would have access to a shower. Suffice it to say, there’s no way we could have sustained that pace. It’s anti-Agile.
A quick search for ideas and opinions about stretch goals makes it very clear that there is no commonly agreed “correct” answer. However, from an Agile perspective stretch goals assigned by outsiders are clearly against the principles of the Agile Manifesto.
The Scrum Guide states:
The number of items selected from the Product Backlog for the Sprint is solely up to the Development Team. Only the Development Team can assess what it can accomplish over the upcoming Sprint.
For emphasis: what it can accomplish – not what it (the Development Team) wants to accomplish, or what it should accomplish, or what it could accomplish if everything goes perfectly. A Development Team should be accomplishing their Sprint plan successfully (all Product Backlog Items done) on a regular basis. Of course, exceptional circumstances may intervene from time to time, but the team should be building trust with stakeholders. Here’s another story:
I had a good friend. We would always go out for coffee together. We just hung out – chatted about life, projects, relationships. Of course, from time-to-time one or the other of us would cancel our plans. That’s just life too. But there came a time when my friend started cancelling more often than not. There was always a good excuse: I’m sick, unexpected visitors, work emergency, whatever. After a little while of this I started to think that cancelling would be the default. I even got to the point where I was making alternative plans even if my friend and I had plans. I got to the point where I no longer trusted my friend. It didn’t matter that the excuses were always good. Trust was broken.
It doesn’t matter why a team fails to meet a goal. It reduces trust. It doesn’t matter why a team succeeds in meeting a goal. It builds trust. Even among team members. A team setting stretch goals is setting itself up for regular failure. Even if the team doesn’t share those stretch goals with outsiders.
Stretch goals destroy trust within the team.
Think about that. When a team fails to meet its own stretch goal, team members will start to look for someone to blame. People look for explanations, for stories. The team will create its own narrative about why a stretch goal was missed. If it happens over and over, that narrative will start to become doubt about the team’s own capacity either by pin-pointing an individual or in a gestalt team sense.
The importance of trust cannot be over-stated. In order for individuals to work effectively together, they must trust each other. How much trust? Well, the Agile Manifesto directly addresses trust:
Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done.
Here is my recent YouTube video about stretch goals… check it out and subscribe to our channel!
Subscribe to our BERTEIG / WorldMindware YouTube channel – we have over 20 more videos planned on the Myths of Scrum.
Regular big up-front planning is not necessary with Scrum. Instead, a team can just get started and use constant feedback in the Sprint Review to adjust it’s plans. Even the Product Backlog can be created after the first Sprint has started. All that is really necessary to get started is a Scrum Team, a product vision, and a decision on Sprint length. In this extreme case, the Scrum Team itself would decide what to build in its first Sprint and use the time of the Sprint to also prepare some initial Product Backlog Items. Then, the first Sprint Review would allow stakeholders to provide feedback and further develop the Product Backlog. The empirical nature of Scrum could even allow the Product Owner to emerge from the business stakeholders, rather than being assigned to the team right from the start.
Starting a Sprint without a Product Backlog is not easy, but it can be done. The team has to know at least a little about the business, and there should be some (possibly informal) project or product charter that they are aware of. The team uses this super basic information and decides on their own what to build in their first Sprint. Again, the focus should be on getting something that can be demoed (and potentially shippable). The team is likely to build some good stuff and some things that are completely wrong… but the point is to get the Inspect and Adapt cycle started as quickly as possible. Which means of course that they need to have stakeholders (customers, users) actually attend the demo at the end of the Sprint. The Product Owner may or may not even be involved in this first Sprint.
One important reason this is sometimes a good approach is the culture of “analysis paralysis” that exists in some organizations. In this situation, an organization is unable to do anything because they are so concerned about getting things right. Scrum is a framework for inspect and adapt and that can (and does) include the Product Backlog. Is it better for a team to sit idle while someone tries to do sufficient preparation? Or is it better to get started and inspect and adapt? This is actually a philosophical question (as well as a practical question). The mindset and philosophy of the Agile Manifesto and Scrum is that trying to produce valuable software is more important that documentation… that individuals and how they work together is more important than rigidly following a process or tool. I will agree that in many cases it is acceptable to do some up-front work, but it should be minimized, particularly when it is preventing people from starting to deliver value. The case of a team getting started without a product backlog is rare… but it can be a great way for a team to help an organization overcome analysis paralysis.
The Agile Manifesto is very clear: “The BEST architectures, requirements and designs emerge out of self-organizing teams.” [Emphasis added.]
Hugely memorable for me is the story that Ken Schwaber told in the CSM course that I took from him in 2003. This is a paraphrase of that story:
I [Ken Schwaber] was talking to the CIO of a large IT organization. The CIO told me that his projects last twelve to eighteen months and at the end, he doesn’t get what he needs. I told him, “Scrum can give you what you don’t need in a month.”
I experienced this myself in a profound way just a couple years into my career as an Agile coach and trainer. I was working with a department of a large technology organization. They had over one hundred people who had been working on Agile pilot projects. The department was responsible for a major product and executive management had approved a complete re-write. The product managers and Product Owners had done a lot of work to prepare a product backlog (about 400 items!) that represented all the existing functionality of the product that needed to be re-written. But, the big question, “what new technology platform do we use for the re-write?” had not yet been resolved. The small team of architects were tasked with making this decision. But they got stuck. They got stuck for three months. Finally, the director of the department, who had learned to trust my advice in other circumstances, asked me, “does Scrum have any techniques for making these kind of architectural decisions?”
I said, “yes, but you probably won’t like what Scrum recommends!”
She said, “actually, we’re pretty desperate. I’ve got over a hundred people effectively sitting idle for the last three months. What does Scrum recommend?”
“Just start. Let the teams figure out the platform as they try to implement functionality.”
She thought for a few seconds. Finally she said, “okay. Come by this Monday and help me launch our first Sprint.”
The amazing thing was that the teams didn’t lynch me when on Monday she announced that “our Agile consultant says we don’t need to know our platform in order to get started.”
The first Sprint (two weeks long) was pretty chaotic. But, with some coaching and active support of management, they actually delivered a working increment of their product. And decided on the platform to use for the rest of the two-year project.
You must trust your team.
If your organization is spending more than a few days preparing for the start of a project, it is probably suffering from this pitfall. This is the source of great waste and lost opportunity. Use Scrum to rapidly converge on the correct solutions to your business problems instead of wasting person-years of time on analysis and planning. We can help with training and coaching to give you the tools to start fast using Scrum and to fix your Scrum implementation.
This article is a follow-up article to the 24 Common Scrum Pitfalls written back in 2011.
[UPDATE: 2015/08/19] I’ve just added a video to the “Myths of Scrum” YouTube series that adds a bit to this:
How much documentation does it take to run a project with ten people working for six months? For some organizations it takes way too much:
This binder (about 3 or 4 inches thick) is all the documentation associated with such a project. In looking carefully at the project, creating the documentation took far more time than the time spent on designing, writing and testing the software. Yet, the documentation does not produce any value. Only the software produces value. The Agile Manifesto, asks us to focus on the outcome (working software) and to make tradeoffs to minimize the means (comprehensive documentation).
The Agile Manifesto asks us to challenge our assumptions about documentation. In many work environments, documentation is an attempt to address some interesting and important needs:
Documentation is usually heavier (more comprehensive) the more the following circumstances exist in an organization:
What if the software itself could address the needs that often documentation is used to address? Let’s look at them in turn:
In my recent training programs as research for this article, I have surveyed over 100 people on one aspect of documentation – code documentation. Every individual surveyed is either currently coding or has a coding background, and every single person had the same answer to a simple scenario question:
Imagine that you have just joined a new organization and you are about to start working as a software developer. One of the existing team members comes up to you and introduces himself. He has with him a piece of paper with a complicated-looking diagram and a full binder that looks to be holding about 250 pages. He asks you, “you need to get up to speed quickly on our existing system – we’re starting you coding tomorrow – would you prefer to go over the architecture diagram with me or would you prefer to review the detailed specifications and design documents.” He indicates the one-page diagram and the binder respectively. Which would you prefer?
(I’ve put up a Survey Monkey one-question survey: Code Documentation Preference to extend the reach of this question. It should take you all of 60 seconds to do it. I’ll post results when I write the next Agile Manifesto essay in a month or two.)
The fact that everyone answers the same way is interesting. What is even more interesting to me is that if you think through this scenario, it is actually almost the worst-case scenario where you might want documentation for your developers. That means that in “better” cases where documentation for developers may not be as urgent or necessary, then the approach of just going to talk with someone is a lot better.
Documentation and Maps
The problem with documentation is the same problem we have with maps: “the map is not the territory” (quote from the wisdom of my father, Garry Berteig). We sometimes forget this simple idea. When we look at, say, Google Maps, we always have in the back of our consciousness that the map is just a guide and it is not a guarantee. We know that if we arrive at a place, we will see the richness of the real world, not the simplified lines and colours of a map. We don’t consider maps as legally binding contracts (usually). We use maps to orient ourselves… as we look around at our reality. We can share directions using maps, but we don’t share purpose or problems with maps. And finally, maps assume that physical reality is changing relatively slowly (even Google Maps).
Many times when we create documentation in organizations, however, we get confused about the map versus the territory.
Agility and Documentation
Of course, code is a funny thing: all code is documentation too. The code is not the behaviour. But in software, code (e.g. Java, ASM, Scheme, Prolog, Python, etc.) is as close as possible to the perfect map. Software is (mostly) deterministic. Software (mostly) doesn’t change itself. Software (mostly) runs in a state absent from in-place human changes to that software. Software (mostly) runs on a system (virtual or physical) that has stable characteristics. The code we write is a map. From this perspective, documentation becomes even less important if we have people that already understand the language(s)/platform(s) deeply.
This essay is a continuation of my series on the Agile Manifesto. The previous two essays are “Value and Values” and “Individuals and Interactions over Processes and Tools“.
This is the story of how the Scrum of Scrums has evolved for a large program I’m helping out with at one of our clients.
Last night I had the honour of giving a talk at the PMI-SWOC. It seemed well received and I really enjoyed the opportunity. The slides from the talk are attached to this post.
There were quite a few people in attendance who were new to Agile and I spent a bit of time talking about the Agile Framework before really getting into the slides of my talk.
On Tuesday Dec. 2, Mishkin Berteig will be speaking about The Agile Enterprise and the five different approaches to implementing Agile at the enterprise level. The talk will also include some details about two frameworks used at the enterprise level: SAFe (Scaled Agile Framework) and RAP (Real Agility Program).
This talk is hosted by the South Western Ontario chapter of the PMI.
You probably already use an Agile Estimation technique such as the Planning Game or the Bucket System, but surprisingly few people understand the underlying principles of Agile Estimation. This lack of understanding often causes confusion or stress for the people who try to use Agile Estimation techniques. The discrepancy between traditional estimation techniques and Agile techniques is large and it is hard to bridge that mental gap without understanding the principles involved. There are three fundamental principles of Agile estimation:
Principle One: Collaborative
Agile estimation methods are collaborative. This means that multiple people work together to arrive at estimates for work in an Agile project or product development effort. Traditional estimation techniques (such as those related to bottom-up or top-down) tend to focus on individuals estimating the work that they are responsible for doing and “trusting” those individual estimates. Collaborative estimation means that most estimation is done by people in formally facilitated meetings where people are present in-person.
Collaborative techniques are generally used where there is some expectation that multiple minds are better than a single mind in discovering some new knowledge or solving a problem. Teamwork and groupwork are based on this concept. This idea of collaboration for problem solving is also applied to Agile Estimation and it has some interesting ramifications.
The most radical consequence of collaborative estimation methods is that there is no possibility to trace a particular estimate for a particular item to a particular individual person. This lack of traceability is important to create a sense of safety on the part of participants in the estimation effort. This safety is necessary to allow participants to be fully honest about estimates even if those estimates conflict with expectations of powerful stakeholders. Another way of stating this principle is that no individual can be punished for a bad estimate.
Many Agile estimation techniques take this principle beyond just mere collaboration to the level of consensus-building techniques where everyone in a group doing estimation work must agree on the final estimate for each and every item being estimated. This strengthens the idea of safety to the point where no participant in an estimation effort can ever say “I didn’t agree with that” and thereby leave other participants “on the hook” for a bad estimate.
Principle Two: Relative Estimation
Imagine you are shown a glass bottle with some water in it. You can see the water sloshing around. Someone asks you, “how much water is in the bottle?” You might, at first, think about the overall size of the bottle and respond by saying “it’s 1/3 full.” Or, if asked to provide a measure in units like millilitres or fluid ounces, you might mentally compare what you see in front of you to some container (e.g. a measuring cup) where you know the quantity. In both cases, you are doing a relative estimate of the amount of water in the bottle. You are comparing the amount of water to a known quantity.
Imagine a counter-example: someone walks up to you with a red pen ask asks you “what is the wavelength of the light being reflected from this red ink?” If you are like most people, you have probably forgotten (if you ever knew) the wavelengths of light in the visible spectrum. You have no basis for comparison. You might take a wild guess, but it is just that. Going back to our relative measure, you might be able to easily say if it is darker or lighter than another red colour, or you might even be able to tell what hue of red it is. But those cases are, again, relying on our inherent ability to see relative differences.
So instead of ignoring this capacity, in Agile estimation techniques, we leverage it. When estimating effort, we start by setting a clear baseline for what we are comparing: another piece of work. The baseline piece of work is often given an “estimate” that is arbitrary and in some non-standard units. For example, it is common to use “points” when estimating the effort for Product Backlog Items.
When doing relative estimates it is very important to ensure we are comparing “apples to apples”. Both the piece of work to be estimated and the comparison piece should both be work items that are not yet done! If you have already completed one of the pieces of work, you have prior knowledge that you don’t have for the work to be estimated.
This last point is subtle, but important. If you have already done something, you know much more about it. If you try to compare to something you haven’t yet done, you will be tempted to assume that the two things will be more similar than they may be when you actually get to work on it. By comparing two pieces of work that you have not yet done, you become much more conscious of the risks of comparing, and that consciousness will help you make better relative estimates.
It is important to note that one side advantage of using relative units for estimation is that it makes it much more difficult to use estimates as a baseline for either measuring performance or for tracking schedule variance, both of which are essentially meaningless in a good Agile environment (which should be almost entirely results-oriented).
Principle Three: Fast
In Agile estimation we don’t care (!!!) about accuracy nor about precision of estimates. Whoa! Why is that? Because estimation is waste. You can’t sell estimates, and estimates don’t affect the “form, fit or function” of the thing you’re building. Therefore, both Agile and Lean concur: do your utmost to eliminate that waste. There are actually lots of Agile practitioners who think estimates are evil (and they have some good arguments!)
In order to do Agile estimation in a maximally non-evil way, we need to make estimation fast! Really fast!!! Many of the Agile estimation techniques allow you to estimate a product release schedule lasting as much as a year in just a few hours given a reasonably well-crafted Product Backlog.
There are really only two modestly good reasons for doing estimation in an Agile project or product:
As a business stakeholder, one can do a simple mental exercise. Ask yourself, “how much money would I be willing to spend to accomplish those two objectives?” Whatever your answer, I hope that it is a very small amount compared to what you are willing to spend on getting results. If not, perhaps you haven’t really embraced the Agile mindset yet where “the primary measure of progress is working software” (the Agile Manifesto).
Bonus Principle: We Suck at Estimating
Most people doing estimation in traditional project management try to measure in units like person-days or dollars. There is no doubt that these units are useful if you can get good estimates. However, most of the people doing estimation are fundamentally and irredeemably bad at it. How do I know? Because they are not wealthy… and have thereby proven that they cannot predict the future. If you can predict the future, even just in limited circumstances (like estimating effort or revenue), you can leverage that to generate almost untold wealth. Given that, it is fruitless and wasteful to try to get better at estimating. Instead, the principles of Agile estimation help us focus our attention on the right things: collaboration, comparative estimates and doing them fast so we can get to the real work, and most importantly: delivering valuable results now. Understanding these principles helps teams overcome many of the struggles they have in using Agile estimation techniques.
I have worked with a lot of people, teams and organizations over the last 8 years helping them to adopt Scrum and I have seen some interesting patterns about where Scrum works well and where it doesn’t work so well. I wanted to share my observations to see if they correlate with what other people are experiencing.
So first off, I want to describe what I mean by Scrum working well:
So now I can describe where I have observed Scrum to work really well:
I have also seen Scrum be inappropriate and not lead to the results I mentioned above:
Scrum can definitely transform the world of product development. It can definitely act as a catalyst to get teams and organizations out of crisis. But that isn’t the whole world of work. I’m also concerned about the idea of using Scrum for general project management. There might be some good practices that are part of Scrum that would also be valuable in general project management (e.g. regular retrospectives, daily team meetings) but that doesn’t make Scrum a general project management framework.
I don’t claim that any of the above observations are “correct”. That’s partly why I am sharing – I would love to have a good discussion here about this because I think it is critical for us as Agilists to be able to answer this question well when we are asked: “what is Scrum good for?” – particularly since Scrum is by far the most popular Agile method.
I would love to hear other people’s observations about where Scrum works well (as I have defined “well” above) vs. where it either is only a modest improvement to existing approaches or where it might even not work at all.
We have an upcoming three-day agile training seminar in Markham September 7 & 8, 2011.
In this unique seminar, we will be offering a practical view of three important Agile methods: OpenAgile – used for general agile project management and agile teamwork including projects and organizations doing any kind of work. Scrum – used for software new product development and IT project management. Kanban – used for teams doing operational work.
This seminar contributes towards three certification programs: the Scrum Alliance’s Certified ScrumMaster program, the OpenAgile Team Member level and the IPMA/PMAC Agile Project Management certification.
Proudly delivered by Berteig Consulting, a Canadian organization since 2004.