Agile Advice: Testing Archives

December 14, 2007

Automated Testing

In agile software development, automated testing plays a big role due to the emphasis on quality. A friend of mine, Tom Cooper, whom I worked with a few years ago at a major capital markets firm, passed these links on to me yesterday. There are some folks at NIST and the Univ. of Texas who are building a tool called FireEye. Here is the press release, here is the pdf presentation.

Posted by Mishkin Berteig at 08:43 AM | |

August 21, 2007

Agile and Lean Defect Tracking

Lisa Crispin has written an excellent article about defect tracking in agile environments. The article is a couple of months old now, but if you haven't read it yet, you definitely should! I particularly like the perspective that Mary Poppendieck offers in the article...

Since Quality is Not Negotiable, and since defects in both XP and Scrum are meant to be dealt with immediately, there is a good argument to be made for doing away with any type of automated tool-based defect tracking.

One team I coached really struggled with this. They were working on the first every agile project in their organization and none of the team members had any prior experience with agile or with test-driven development. It was a small project, but management wanted to do things "right". They had a team room, a cross-functional team, short iterations, etc. etc.

One member of the team was a person who was their Quality Assurance expert. This person was extremely capable: able to write excellent test scripts (in Excel) and able to think carefully about quality. However, the organization was not set up to support an agile approach to quality: QA folks were measured on how many defects they found and logged in their defect tracking system.

Since the team had decided to use test-driven development, they soon found that they had an odd question: when does incomplete work become defective work that needs to be logged? In other words, since the tests are written first and they start out by failing, and since this is normal and happens for every single feature, exception condition and even method of code, tracking all these as "defects" would be meaningless. So how do you decide that something is a defect instead of test-driven work that just isn't yet making the tests pass?

After a great deal of discussion and confusion and consternation, we finally agreed to a simple and effective practice: inside of an iteration, any issues or problems that come up are written on the whiteboard. The whiteboard needs to be clear by the end of the iteration. If it is not, then the items that are still there are logged into the defect tracking system. These items also become top priority for the next iteration. As well, any items which had non-passing tests by the end of the iteration were also put into the defect tracking system and became top priority for the next iteration.

This helped... and we only ended up with at most one or two items at the end of each iteration that could be logged... and often there were no known defects.

This was legitimately painful for the QA person whose defect logging rates went way down. Fortunately, since this was a pilot project, it was easy to explain to management what was going on. This is a very good example of how a management practice that makes a lot of sense in a non-agile environment becomes a huge obstacle in an agile environment.

Posted by Mishkin Berteig at 07:46 PM | |

January 02, 2007

Top Ten Most Popular Entries from 2006

If you are new to Agile Advice, these are not just some of the most popular articles, they are also some of the best! Take a look around; you will find ideas to inspire you, challenge your thinking and maybe that one little thing that will make the difference in applying agile methods in your situation.

1. How Two Hours Can Waste Two Weeks - 25,617 unique views
This little hypothetical story by Dmitri Zimine was very popular on Reddit, and Joel on Software ranted a bit about it.

2. The Case for Context Switching - 2,936 unique views
My rebuttal to Joel's rant. Goes well with Dmitri's article. Emphasizes the idea of building trust in an organization.

3. The Qualities of an Ideal Test - 1,579 unique views
Six qualities that will help make your tests as valuable as can be. Similar to the ACID properties of databases or the INVEST properties of user stories.

4. The Pros and Cons of Short Iterations - 1,555 unique views
A few words that will help you decide how long to make your iteration length. This is an important decision, and most teams and organizations don't know the factors involved.

5. Five Signs of Trouble in an Iteration - 1,489 unique views
A good howto on using burndown charts to discover problems in an iteration.

6. Seventeen Tips for Iteration Planning - 1,427 unique views
A good list of hints and tips that can make the difference between struggling with iteration planning and having it go smoothly and effectively. This is a key part of the Agile Work process, so getting good at it is a high priority for any team new to Agile Work.

7. Change is Natural - "Embrace Change" - 1,397 unique views
A short riff on the universality of change. Also introduces the idea of the "horizon of predictability".

8. The Art of Obstacle Removal - 1,287 unique views
This is a good reference article on types of obstacles and methods for removing them... a critical practice for Process Facilitators.

9. The Seven Core Practices of Agile Work - 1,285 unique views
Agile Work is really quite simple. This is a concise list of the practices that make up this effective methodology.

10. Interview with Alistair Cockburn - Agile and House Renovations - 902 unique views
Applying agile methods to home and garden renovations! Learn a bit about how this luminary of the agile world has taken agile practices out of the software world and into the hardware world.

Posted by Mishkin Berteig at 06:32 PM | |

December 22, 2006

Technical Debt

Last night I was thinking more about the analogy of technical debt. In this analogy, design and quality flaws in a team's work become a "debt" that must eventually be paid back. This analogy is fantastic because it can be taken just a little bit further to understand just how bad defects are...

Debt is sometimes useful. Financial debt can be used for risk reduction, investment, and emergencies. But it can also cause problems. Too much debt becomes hard to manage. If debt maintenance costs more than revenue (minus other expenses), then you're going down!

Technical debt is a little different than financial debt.

Suppose I went into a boardroom full of high-power executives for some large company. (Never mind how I got there.) And I pitched this fabulous idea: I'll give them a bunch of money for them to use for operations, expansion, whatever. All they have to agree to is that I choose the interest rate when I ask for repayment... trust me!

I would be thrown out of that room so fast!

That is what we do when we encourage teams to take on technical debt.

There is no way to know the interest rate on a defect. Part of the cost of a defect is obvious: how much time and materials will it take to repair the immediate problem. (Although even that is often hard to measure.) But there are also lots of non-obvious and probably non-measurable costs. How much effort will it take to get to the root cause of the defect so that it doesn't occur a second time? How much will it affect our "goodwill" and thus reduce further and repeat sales? How much will the existence of one defect hide the existence of other defects (with their own costs)? How much will the defect demoralize the team and increase staff turnover or reduce productivity? How much of an opportunity will the defect create for competitors? How much will the defect increase maintenance and support costs?

In other words, every time someone asks a team to let quality slide, they are asking the team (and the organization) to take on debt with an unknown interest rate. Which is lunacy.

September 05, 2006

The Seven Core Practices of Agile Work

Agile Work consists of seven core practices. These practices form a solid starting point for any person, team or community that wishes to follow the Middle Way to Excellence.

Self-Organizing Team

Any group of people that wish to be an Agile Team need to take the initiative to determine for themselves how they are going to work (process) and how they are going to do the work (product). The term "team" really applies quite broadly to any size group of people that are working together towards a common goal.

Teams go through stages of development as they perform their work. The most important result of team development is the team itself, and not the specific skills and abilities that the individuals learn.

If the team is part of a broader organization, that organization must give the team the authority, space and safety to learn to be self-organizing. The organization's leadership is responsible for determining the "why?", some constraints on "how?", and then letting the team respond to the need as best as it can.

Also Known As: Whole Team (Extreme Programming), Cross-Functional Team (business management).

Deliver Frequently

Agile Work uses short fixed periods of time to frame the process of delivering something of value. Each of these iterations or timeboxes is structured so that the team or group actually finishes a piece of work and delivers it to stakeholders. Then, the team builds on what has previously been delivered to do it again in the same short amount of time.

The sooner that valuable results can be delivered, the more value can be obtained from those results. This extra value is derived from opportunities such as earlier sales, competitive advantage, early feedback, and risk reduction.

There is an explicit tradeoff: the shorter the time to delivery, the smaller the piece of value will be. But, like investing in one's retirement account, the earlier you start, even with small amounts of money, the better off you are in the long run.

Also Known As: Sprint (Scrum), Iteration (Extreme Programming), Timeboxing (generic), Time Value of Money (accounting).

Plan to Learn

Every type of work is governed by a Horizon of Predictability. Any plan that extends beyond this horizon of predictability is bound to fail. Agile work uses an explicit learning cycle tied in with the planning of work to accomodate this inevitable change.

First, a goal is required. This goal can be long-term. Teams using Agile Work then create a queue of work items to be done in order to reach this goal. Each iteration, some of these items are selected, finished and then the queue is adjusted. The changes in the work queue are based on external factors, and learning that the team does as it goes.

One of the most effective methods for the team to learn about how it is doing its work is the retrospective. After each delivery of results, the team holds a retrospective to examine how it can improve.

Also Known As: Inspect and Adapt (Scrum), Kaizen (Lean), Adaptive Planning (generic).

Communicate Powerfully

A team needs to have effective means of communicating, both amongst team members and also to stakeholders. To Communicate Powerfully, a team needs to prefer in-person communication over distributed communication. Synchronous over asynchronous communication. High-bandwidth over low-bandwidth communication. Multi-mode communication over single-mode communication.

The results of failing to communicate powerfully include wasted time for waiting, misunderstandings leading to defects or re-work, slower development of trust, slower team-building, and ultimately a failure to align perceptions of reality.

The single most effective means to communicate powerfully, is to put all the team in a room together where they can do their work, every day for the majority of the work time.

Some types of work do not lend themselves to this approach (e.g. creating a documentary video), but every effort should be made to improve communication.

Also Known As: Visibility (Scrum), Whole Team and Team Room (Extreme Programming), War Room (business management).

Test Everything

Defects are one of the most critical types of waste to eliminate from a work process. By testing everything, by driving all the work of a team by creating test cases to check the work, a team can reach extremely high quality levels. This ability to prevent defects is so important that only an executive level decision should be considered sufficient to allow defects into a work process. Quality is not negotiable.

In Agile Work, removing a defect is the only type of work that takes priority over any new features/functionality/production. If the end result desired is to maximize value, then removing defects is an important means to that end.

A team has an ethical duty to discover new ways to effectively test their work. This can be through the use of tools, various feedback mechanisms, automation, and good old problem-solving abilities.

Also Known As: Canary in the Coal Mine (Scrum), Test-Driven Development (Extreme Programming), Defects per Opportunities (Six-Sigma).

Measure Value

Since Reality is Perceived, it is important for an agile team and organization to have a clear method of describing and perceiving what is important for the organization. Measuring value is a critical method for describing and perceiving what is important.

A single metric can be used to drive all the measurement and goal-setting and rewards in an organization. All other measurements are secondary and must be treated as such: limited in use and temporary.

There are many things which are easier to measure than value. It is often easy to measure cost, or hours worked, or defects found, or estimate vs. actual... etc. However, all of these other measurements either implicitly or explicitly drive sub-optimal behavior.

Also Known As: Measuring Results (Scrum), ROI (business management), Economic Driver (Good to Great), Running Tested Features (Extreme Programming).

Clear the Path

Everyone in an organization using Agile Work takes responsibility for clearing the path, removing the obstacles that prevent work from being done effectively. Clearing the Path doesn't just mean expedient, quick fixes to problems, but rather taking the time to look at an obstacle and do the best possible to remove it permanently so that it never blocks the path again.

In the Agile Work method, the Process Facilitator is the person who is responsible for tracking obstacles and ensuring that the path is cleared. To do this, the Process Facilitator maintains a Record of Obstacles.

Clearing the Path is sometimes painful work that exposes things we would rather not deal with. As a result, it is critical that people build their capacity for truthfulness and work to develop trust amongst each other. Building a capacity for truthfulness is not something that can be done by using an explicit process.

Also Known As: Removing Obstacles (Scrum), Stopping the Line (Lean).

Remember also, that these practices must always be viewed and implemented in the context of the Agile Axioms. These axioms provide a check to ensure that the practices are not being applied blindly, but rather applied appropriately to the given situation.

Posted by Mishkin Berteig at 09:09 AM | |

August 18, 2005

Harvard Business Review Article

I highly recommend this article on Collaboration Rules. Great stuff in there about developing teams, developing organizations and how important communication and trust are to doing so. The article draws examples from and compares the open-source development and maintenance of the Linux kernal and the operation of the Toyota Production System.

Posted by Mishkin Berteig at 10:48 PM | |

May 19, 2005

Test-Driven Work

In an agile environment, all work done needs to be directly related to the needs of stakeholders. Stakeholders request or “pull” work from the team, and they do this by defining prioritized work packages. The team needs some way to know when they have completed a work package, so both work packages and iteration tasks need to have testable acceptance or success criteria. The team collaborates with the stakeholders to determine what needs to be done to successfully complete a work package.

Based on the acceptance criteria, tests are described or created. Ideally, these tests are created before or simultaneously to any work that is done on the work package or task. Any work done is done only to make the tests succeed – no speculative (wasteful) work should be done. The team members should carefully avoid the belief that they can predict work that needs to be done if there is no test for that work.

Tests can be informal, formal or even automated depending on the environment and the type of work being done. Tests can be questions or measurements and their expected results. A test can also be a procedure to follow and the results of that procedure. If the environment supports it, automating tests can be an excellent investment for reducing waste. In an ideal environment and work domain, tests can fullfil all the attributes of an ideal test.

Test driven work has two solid benefits: it helps ensure close collaboration between the team and the stakeholders, and it helps eliminate the waste of unnecessary work. Thus it supports the three agile work disciplines of Empower the Team, Amplify Learning and Eliminate Waste.

In software development, where Test-Driven Work is very sophisticated, there are a number of excellent testing tools and resources.

Posted by Mishkin Berteig at 06:01 PM | |

May 17, 2005

The Qualities of an Ideal Test

These six qualities of tests describe how to make a test as effective and as useful as possible. The qualities are similar in style to the INVEST qualities of user stories - but they don't form a nice acronym.

Decisive: The test contains all the information required to automatically determine success or failure. Manual inspection of test results is not necessary to determine success or failure. The test is expressed in a way that produces a pass/fail answer rather than a numerical or qualitative result. Decisive tests are often expressed as assertions.

Valid: The test produces a result that matches the intention for the work artifact under test. Test failure indicates failure in the artifact under test and test success indicates correct operation or configuration of the artifact under test. Simply put: the result of a test should be true.

Complete: The test contains all the information it needs to run correctly with a given test harness and work artifact under test. The test performs all activities and provides all the data necessary. The test does not require any input outside of itself in order to run.

Repeatable: The test always gives the same results if the test harness and the artifact under test are the same. The test is created in such a way that the result is deterministic. Even if the system under test is not deterministic, the test should be created to account for that (possibly with statistical analysis) and produce a deterministic result.

Isolated: The test result is not affected by other tests run before it nor does a test affect the results of tests run after it. The test and the test harness work together to clean up after every test is run. A collection of tests can be run in any order and always produce the same results. Any test that depends upon the results or side-effects of a previous test is not isolated.

Automated: The test requires only a start signal in order to run to completion in a finite amount of time. No manual intervention is required after the test is started. Tests that are automated can be put together into a test suite and run one after another without human intervention. Automation of tests requires the creation of a test harness system.

Update 20060106

I've added a little bit more detail to the above qualities. I also wanted to say a few words about my experience with testing:

I first started doing test-driven development almost seven years ago after reading the book Refactoring by Martin Fowler. I picked it up in Windsor Ontario while I was driving to San Francisco for my first contract position as an independent. I had to stay the night there due to the immigration office being closed so I read a lot of the book in my motel room that night. By time I got to California four days later, I was totally convinced that I should be using JUnit and refactoring for everything I did. Fortunately my project was ammenable to this approach. Over the next four months I built a Java JMS layer on top of Tibco Rendezvous. In the process I discovered methods for doing unit tests for asynchronous multi-threaded distributed functionality. And my tests satisfied all the above qualities. I delivered code with 100% test coverage and zero defects discovered until more than two years later when a very rare and obscure deadlock issue was found. Suffice it to say, I was convinced from a quality perspective.

But there is more. To build tests that follow all the above qualities, you need code that is testable. I have come to believe that testability is the most important architectural attribute of code for most software. The implications of having testable code include code that is easily changed, that is verifiable, and that is easy to understand. Refactoring plays a big role here too.

For getting a basic but very practical sense of test driven development, I strongly recomment Test Driven Development: By Example by Kent Beck.

I have also written a couple of articles recently about quality:

Quality is Not Negotiable in which I talk about how to handle defects - by always treating them as top priority work items.

and

Technical Debt in which I expand on this common analogy between lack of testing and debt to show that technical debt is even worse than financial debt.

Posted by Mishkin Berteig at 06:48 PM | |

Agile Advice

Thoughts and experiences for practitioners of Agile Work, the Middle Way to Excellence.