Tag Archives: knowledge

Agility: Knowledge Generation and Beauty

Compromised Agility

The authors of the Manifesto for Agile Software Development expound a set of values and principles that define “Agile Software Development”. These values and principles, written in 2001, became the focal point of a revolution in how software developers work.

In the last several years, that revolution has spread beyond software development to encompass other aspects of technology, and beyond technology into operations, management, engineering, business development, sales, marketing and even outside of for-profit organizations into education, health, government, community and charitable organizations. Originally written in the context of software, we can easily generalize the values and principles to other types of work. For example, the Manifesto refers to valuing “working software over comprehensive documentation.” We can easily generalize this to a more abstract statement that we value “results over bureaucracy.” The other values and principles can be similarly abstracted.

As the revolution has spread, unfortunately, the values and principles have also become compromised or selectively applied. Perhaps most obvious is how Agile Lifecycle Management tools such as Jira have often been substituted in place of the first value of the Manifesto: “we have come to value individuals and interactions over processes and tools” [emphasis added]. The irony of this substitution seems to be lost on those selling and buying these tools. Many other compromises or selective applications are common, and they are usually unique to each organization’s circumstances.

This evolution of the application of the Manifesto’s values and principles can be understood negatively or positively. I’m an optimist; yet we need to look at the negative, critical side before we can see the positive.

Cargo Cult Agility / No True Agilist

There are two common ways to understand the failure of organizations to do “true Agile”. Both are negative in the sense that they are criticisms without a reasonable solution to help an organization out of the situation into a better situation. As a consultant seeing many organizations, as a trainer hearing about many organizations, and as an active member of the global community of Agilists, I hear these criticisms quite regularly.

The first approach to understanding this partial agility is by comparison to the cargo cult mentality:

A cargo cult is a millenarian movement first described in Melanesia which encompasses a range of practices and occurs in the wake of contact with more technologically advanced societies. The name derives from the belief which began among Melanesians in the late 19th and early 20th century that various ritualistic acts such as the building of an airplane runway will result in the appearance of material wealth, particularly highly desirable Western goods (i.e., “cargo”), via Western airplanes.

In this approach to understanding an organization’s failure to embrace true Agile, the critic asserts that the leaders and employees of the organization do not really understand true Agile and are only practicing the obvious ritualistic aspects such as daily stand-ups (from Scrum), note cards on a wall (from Kanban), or user stories (from Extreme Programming). This criticism has some merit: many people are told to “do Agile” without proper training and coaching to understand the theory or the contextual applicability of various practices. The critic continues to compare this approach to Agile as a belief in magic: that the benefits of Agile can be gained through an application of the rituals without an understanding of, and more importantly adoption of the values and principles.

The “cargo cult” criticism does not offer a solution. When asked, the critics themselves will say, effectively, “well, you just need to really understand it!” This criticism also suffers from the inherent notion of the superior position of the critic: “I understand it… you don’t.” Not particularly helpful, especially for the staff in an organization who depend on executives and other leaders to support true Agile. And, not particularly helpful for the executives who often do not have the skill to support such a deep change.

The second common approach to understanding the failure of organizations to achieve real agility is the “No True Scotsman” comparison. This is a bit more challenging to describe because it is actually a criticism of other critics. Wikipedia describes No True Scotsman this way:

No true Scotsman or appeal to purity is an informal fallacy in which one attempts to protect a universal generalization from counterexamples by changing the definition in an ad hoc fashion to exclude the counterexample. Rather than denying the counterexample or rejecting the original claim, this fallacy modifies the subject of the assertion to exclude the specific case or others like it by rhetoric, without reference to any specific objective rule (“no true Scotsman would do such a thing”; i.e., those who perform that action are not part of our group and thus criticism of that action is not criticism of the group).

The starting point of this criticism is actually the reciprocal of the cargo cult criticism: the critic conflates true agility with the compromise happening at an organization and then accuses agility of being a failure. This criticism is often brought up in the following style of discussion:

Person A: Agile sucks because Big Corp is trying Scrum but it is really just an excuse for executives to micro-manage every little bit of work.

Person B: But Scrum and Agile are against micro-management! Big Corp isn’t really doing true Agile.[NOTE: this is the cargo cult criticism in brief.]

Person A: That’s just an excuse. You’re using the no true Scotsman argument and it’s a logical fallacy. Agile is what people are actually doing, therefore Agile sucks.

Person B: !

Interestingly, this argument style is often used between competing brands of agility. Leaders of both the Scrum and Kanban approaches to agility are well known for this approach to argument, particularly about each other’s chosen approach: “method X doesn’t work, as clearly seen by all the organizations doing X badly, therefore you should try my method Y which does work.” Again, this is ironic given the first value of the Manifesto for Agile Software Development “…over processes….”

Like the cargo cult criticism, the no true Agilist criticism does not offer a solution, other than reverting to a non-Agile approach to work (or more rarely, another approach that suffers the same imperfect implementation in organizations). And, like the cargo cult criticism, there is some truth in the no true Agilist view: legitimately, many organizations are doing a very very poor job of applying the values and principles (and associated practices). The conclusion that Agile sucks and therefore we shouldn’t even be trying is forgivable. However, the people online proclaiming this problem tend to be loud in their conclusion: let’s go back to our halcyon bygone days where the wind was fresh, the sun was bright, and we were all bucolically happy with our defined roles, our rigid processes, and our tools we could blame for every failure of our efforts (ironic hyperbole intended).

Both criticisms, reciprocal as they are, leave a gap. They don’t offer a full explanation of what is happening, nor do they offer a positive path to improvement. For that, we need to change our perspective just a bit.

Knowledge Generation and the Benefit of Time

I have been working with Agile methods for over 20 years. As a programmer and then enterprise architect, I adopted agile methods relatively early – even before the term “Agile” was applied to software development and these methods were referred to as “lightweight” methods. I mention this not to tout my expertise or even my experience, but rather my perspective. I’ve “been around” enough to know with a fair degree of certainty the following important points:

  1. There is no large organization that has successfully “transformed” from a non-Agile state to an enterprise-wide Agile state. By “large”, I mean at least 5000 total employees.  By “transformed” I mean that it grew up using traditional techniques and has fully switched over to Agile management techniques throughout the total employee population. And by “successfully” I mean that it has sustained the use of Agile management techniques at the organizational level through at least one boom/bust economic cycle, and one change of C-Level leadership. (If you are reading this, and you know of such an organization, I would be excited to hear about it!)
  2. Large organizations that have high levels of agility typically started as small organizations with high levels of agility. Google, Amazon and Facebook come to mind. Their level of agility includes a high level of technical agility in addition to their management agility. I’m not sure, but I suspect that these two types of agility go hand-in-hand; certainly the Manifesto for Agile Software Development suggests they do go together.
  3. The only proof of transformation is in the sustainability of that transformation through existential crisis.
  4. Partial or compromised agility is the only kind of agility that is, so far, successfully sustained in a few rare cases of large organizations. Capital One is an example of this. They have been trying to adopt Agile methods and approaches since 2001. They have tried many techniques, and had many ups and downs in their journey. In 2017 at the Lean Kanban North America conference, two senior Agile coaches from Capital One asserted that they still have a long way to go for full enterprise agility – after working at it for 17 years already.

How does this compare to other management philosophies? The history of the Toyota Production System and Lean manufacturing, at least 50 years in the making so far, has shown us that revolutionary changes in the way work is done, can take many decades to become normalized. In the 1980’s and 90’s many organizations adopted “lean” as a management fad.  We saw a swift rise and then fall in the popularity of lean. But the core principles and ideas of lean survived, and have continued to spread throughout many industries. Now, many organizations are “culturally” lean: they won’t revert to other methods of working even in crisis situations… and many others are not yet there.

With a timeline of 50+ years, perhaps we can consider our efforts to transform organizations to a greater level of agility in a more positive light.  Human society at large is learning about agility though many many experiments run in thousands of organizations. Sometimes these experiments are motivated by wise and considered thought, but often, they are motivated by the faddish popularity of Agile methods such as Scrum, SAFe, and Kanban. Regardless of the motivation, the compromises organizations are making as they attempt greater levels of agility are part of a larger process encompassing all of human society in which knowledge generation is the primary outcome.

There is still one more problem to address: that individuals and organizations continue to try to improve agility even when they have experienced it done badly or seen it fail to take hold. One of my favourite authors and a prominent figure in the Agile community is Ron Jeffries. In the last couple of years, he has started to call out “Dark Scrum” as a blight in organizations that is causing suffering. I believe there is also “dark kanban”, “dark SAFe”, “dark extreme programming”, etc. These dark implementations of the various paths to agility, aren’t actually paths to agility for the people, teams and organizations implementing them. So again, why do people keep coming back to these methods?

Motivation to Try Again and Beauty

Let’s return to the Manifesto for Agile Software Development and here quote the four values. If you have read them before, even many times, I encourage you to read them again, slowly, and savour them:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

I would like to assert that anyone who has read these values, and further, read the principles behind the manifesto has been attracted to the inherent beauty of these ideas. This beauty is the source of the motivation to try again. This beauty is the source of the influence of the manifesto. This beauty is the reason why so many want to “own” it through the creation of their own brands, methods, schemes, and promotions. And, this beauty, while it is constantly struggling against corrupting influences, is powerful enough to inspire people to come back to it even when they have seen the dark side of agility.

Real agility is a culture founded on the beauty that inspires these principles. It’s not any particular method, it’s not a formula, it’s not merely a set of laudable or effective practices. Rather, this beauty is inspired by the Spirit of the Age in which we live. As individuals, teams and organizations, we will rarely live up to that beauty, rather we will experience it in moments, greater or lesser, strung together by our efforts to increase humanity’s collective knowledge. And as that knowledge grows, our patience for the partial, incomplete agility of many organizations will also grow for it is the source of our new-found knowledge.

Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!

The Agile Manifesto – Essay 3: Working Software over Comprehensive Documentation

How much documentation does it take to run a project with ten people working for six months?  For some organizations it takes way too much:

Photo of heavy documentation for software project

This binder (about 3 or 4 inches thick) is all the documentation associated with such a project.  In looking carefully at the project, creating the documentation took far more time than the time spent on designing, writing and testing the software.  Yet, the documentation does not produce any value.  Only the software produces value.  The Agile Manifesto, asks us to focus on the outcome (working software) and to make tradeoffs to minimize the means (comprehensive documentation).

The Agile Manifesto asks us to challenge our assumptions about documentation.  In many work environments, documentation is an attempt to address some interesting and important needs:

  • Knowledge sharing among stakeholders and the people working on a project.
  • Knowledge sharing across time as people come in and out of a project.
  • Verification and traceability for contracts or other compliance needs.
  • Decision-making and analysis for business and technical problems.
  • Management oversight and control.
  • Various aspects of individual accountability.

Documentation is usually heavier (more comprehensive) the more the following circumstances exist in an organization:

  • Geographical distribution of people.
  • Lack of trust between people, departments or organizations.
  • Regulated work environments.
  • Depth of management hierarchy.
  • Number of people directly and indirectly involved.
  • Knowledge and skill sets highly segregated between people.
  • Culture of respect for written texts.

Working Software

What if the software itself could address the needs that often documentation is used to address?  Let’s look at them in turn:

  • Knowledge sharing among stakeholders and the people working on a project.
    If the software is functional at all stages, as supported by Agile methods such as Scrum and Extreme Programming, then the software becomes an effective representation of the knowledge of all the people who have participated in building it.
  • Knowledge sharing across time as people come in and out of a project.
    Software that is technically excellent is often easier to understand for people who are new to it.  For example, excellence in user experience and design means new users can get up to speed on software faster.  Use of good design patterns and automated testing allows new developers to understand existing software easily.
  • Verification and traceability for contracts or other compliance needs.
    Test-driven development (code) and specification by example (scripting and code) are forms of traceable, executable documentation that easily stay in-sync with the underlying software system.
  • Decision-making and analysis for business and technical problems.
    In particular, diagrams can help a great deal here.  However, electronic tools for creating such diagrams can be slow and awkward.  Consider the practice of Agile Modelling (basically using a whiteboard and taking photos) as a good alternative to precise technical diagramming if you are doing problem-solving.
  • Management oversight and control.
    Reports and metrics drive much of the traditional documentation in an organization.  Simplifying reports and metrics often leads to a clearer picture of what is going on, reduces the opportunities to “game” the system, and always results in lower levels of documentation.  As well, some reports and metrics can be generated 100% through automated means.  All that said, the fundamental premise in the Agile manifesto is that management should base decisions on what is actually built – the “Working software” by looking at it and using it.
  • Various aspects of individual accountability.
    If you really need this, a good version control system can give you the information for this.  Sign-offs and other types of accountability documentation are typically just waste that doesn’t actually help in process improvement.  Most people who are in high-compliance environments already have licenses and/or security clearances that provide this accountability.  If you software is working, however, then this isn’t even a concern as trust is built and bureaucracy can be reduced.

In my recent training programs as research for this article, I have surveyed over 100 people on one aspect of documentation – code documentation.  Every individual surveyed is either currently coding or has a coding background, and every single person had the same answer to a simple scenario question:

Imagine that you have just joined a new organization and you are about to start working as a software developer.  One of the existing team members comes up to you and introduces himself.  He has with him a piece of paper with a complicated-looking diagram and a full binder that looks to be holding about 250 pages.  He asks you, “you need to get up to speed quickly on our existing system – we’re starting you coding tomorrow – would you prefer to go over the architecture diagram with me or would you prefer to review the detailed specifications and design documents.” He indicates the one-page diagram and the binder respectively.  Which would you prefer?

(I’ve put up a Survey Monkey one-question survey: Code Documentation Preference to extend the reach of this question.  It should take you all of 60 seconds to do it.  I’ll post results when I write the next Agile Manifesto essay in a month or two.)

The fact that everyone answers the same way is interesting.  What is even more interesting to me is that if you think through this scenario, it is actually almost the worst-case scenario where you might want documentation for your developers.  That means that in “better” cases where documentation for developers may not be as urgent or necessary, then the approach of just going to talk with someone is a lot better.

Documentation and Maps

The problem with documentation is the same problem we have with maps: “the map is not the territory” (quote from the wisdom of my father, Garry Berteig).  We sometimes forget this simple idea.  When we look at, say, Google Maps, we always have in the back of our consciousness that the map is just a guide and it is not a guarantee.  We know that if we arrive at a place, we will see the richness of the real world, not the simplified lines and colours of a map.  We don’t consider maps as legally binding contracts (usually).  We use maps to orient ourselves… as we look around at our reality.  We can share directions using maps, but we don’t share purpose or problems with maps.  And finally, maps assume that physical reality is changing relatively slowly (even Google Maps).

Many times when we create documentation in organizations, however, we get confused about the map versus the territory.

Agility and Documentation

Of course, code is a funny thing: all code is documentation too.  The code is not the behaviour.  But in software, code (e.g. Java, ASM, Scheme, Prolog, Python, etc.) is as close as possible to the perfect map.  Software is (mostly) deterministic.  Software (mostly) doesn’t change itself.  Software (mostly) runs in a state absent from in-place human changes to that software.  Software (mostly) runs on a system (virtual or physical) that has stable characteristics.  The code we write is a map.  From this perspective, documentation becomes even less important if we have people that already understand the language(s)/platform(s) deeply.

This essay is a continuation of my series on the Agile Manifesto.  The previous two essays are “Value and Values” and “Individuals and Interactions over Processes and Tools“.


Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!

Foundations of Excellence

I was thinking about the concept of becoming excellent at something.  My son is a budding artist.  He and I had a conversation a few months ago about talent or aptitude.  I said to him that I felt that aptitude is only latent: you need to put effort into something in order to expose your talent.  He was concerned that he didn’t have any aptitude because he had to work so hard to become better at drawing.  I compared him to myself and my brother, Alexei: when we were growing up, we both put a lot of effort into drawing.  Quickly, I fell behind my brother in skill.  He clearly had aptitude.  But he also put in a lot of effort into exposing that talent.  I was reminded of all this because my son is struggling with math.  He has aptitude, but he hasn’t put much effort into it.  I was wondering why?

Then I realized that aside from aptitude and effort, two more things need to be in place to achieve excellence: willingness and confirmation.

Willingness is the internal drive, usually motivated by an unconscious set of factors, but sometimes also coming from a strong conscious decision.  Willingness can come from unusual combinations of circumstances.  I was extremely willing to learn mathematics in my youth.  This came from two experiences.  One, in grade 2, was when my teacher told me that I shouldn’t be learning multiplication (my dad had taught me while on a road trip).  I was upset that I shouldn’t be able to learn something.  Then, in grade 3, I had a puppet called Kazir (a gift from my babysitter who told stories about space adventures with Azir and Kazir the Baha’i astronauts).  I brought Kazir to school one day and while doing math problems, I pretended that Kazir was helping me.  Suddenly I found math easy.  These two events plus a few others contributed strongly to my desire, my willingness to learn math.

Confirmation is the set of environmental factors that helps keep us on a path of learning.  These environmental factors are sometimes mimicked in the corporate world with bonuses and gamification, but these are really distant shadows of what confirmation is really about. Confirmation is when the stars align, when everything seems to go right at just the right time, when the spirit inspires and moves you and the world to be, in some way, successful.  The trick about confirmation is that success is not usually about monetary success.  It’s usually about social, relational or even sacrificial success.  As an example, when I was in grade 7, I was chosen with a small group of people in my class to do accelerated math studies.  This was a great honour for me and was a confirmation of my interest in math.

In organizational change, and in particular in changing to an Agile enterprise, we need to be aware that excellence requires that these four factors be in place.  Aptitude is, to some degree, innate.  We can’t trick people to have aptitude.  If someone is just fundamentally bad at a certain thing, despite vigorous educational efforts, then that person likely doesn’t have the aptitude.  Effort is about both having time and resources, but also, then about willingness.  And willingness, in turn, can only be sustained with confirmation.  Too much discouragement will break a person’s willingness.  The Agile enterprise requires a great number of skills and abilities that are not normally part of a person’s work environment prior to attempting to adopt Agile.  Keeping these four things in mind can help people in an organization to reach excellence in Agility.

Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!

Project Lessons Learned vs. Sprint Retrospective – 17 Points of Comparison

Another fantastic article by Mike Caspar: Sprint Retrospective vs. Lessons Learned (a Generalization)

Mike says:

Consider reviewing these differences in your environment to determine if you are getting benefit from your Sprint Retrospectives and following their intent.


Here are a few other Agile Advice articles about Retrospectives.

Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!

Amplify Learning

Learning is the result of both encountering new experiences and deliberate experimentation. Learning creates new knowledge, increased volition and improved action.


Knowledge Volition Action

Some of people’s best learning comes from “failure”. An essential component of learning is feedback.

Learning and feedback can be amplified in several ways. Provide opportunities for learning through books, training courses, coaching, deliberate exposure to diverse work, and deliberate experimentation. Frequently ask the questions “why?” and “how?” and answer them honestly and deeply. Increase the level and quality of communication among the stakeholders and team members. Inspect work in progress frequently or even continuously. Learning accelerates greatly if a culture of learning is created where individuals feel free to experiment and take initiative even on critical aspects of the work.

Learning and feedback support all three agile principles. People become more effective creators as they learn. People are better able to adapt to and embrace change as they learn. And a person’s span of perception increases as they learn. Any increase in learning or feedback leads to an increase in the manifestation of the principles.

Learning and feedback can be amplified rapidly, but an empowered team is necessary to effectively take advantage of this amplification. If a team is not empowered, then rapid learning can lead to frustration. Amplified learning and feedback result in excitement, enthusiasm and playfulness, rapid problem solving, high quality work results and satisfied stakeholders.

An excellent analysis of learning at a social or group level is presented in Progress and Its Problems by Larry Laudan. In this very well written book, Laudan takes a look at the history and philosophy of science and develops a model for learning that is applicable to the development of agile methods.

Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!