I’ve just submitted a proposed session to the Agile 2010 web site:

Test-Driven Development for the iPhone, iPod and iPad

Please go to the site and let me know in the comments there if you have any suggestions about the proposal!

Isráfíl Consulting is finally prepared for the first series of its Agile Software Engineering Practices training courses. This series is offered in partnership with Berteig Consulting who are graciously hosting the registration process. Their team has also helped greatly in shaping the presentation style and structure of the course. The initial run will be in Ottawa, Toronto (Markham), and Kitchener/Waterloo.   

Topics covered will include Test Driven Development (TDD), testability, supportive infrastructure such as build and continuous integration, team metrics, incremental design and evolutionary architecture, dependency injection, and so much more. (This course won’t present the planning side of XP, but covers many other aspects common to XP projects) It makes a great complement for training in Agile Processes such as XP, Scrum, or OpenAgile. The overview slide presentation is available for free download from the Isráfíl web site.

The courses are scheduled for:

• 2009-02-05: 2 Day Agile Software ENGINEERING Practices Training – Kichener/Waterloo, Ontario (15 Spots)
• 2009-02-12: 2 Day Agile Software ENGINEERING Practices Training – Markham/Toronto, Ontario (15 Spots)
• 2009-02-19: 2 Day Agile Software ENGINEERING Practices Training – Ottawa, Ontario (15 Spots)

The course is $1250 CAD per student, and participants receive a transferrable discount of $100 CAD for other training with Berteig Consulting as a part of our ongoing partnership. I initially prototyped this course in Ottawa this December, and am very excited to see this through in several locales. Class size is limited to 15, so we can keep the instruction style more involved. The above schedules are linked to Berteig Consulting’s course system and have registration links at the bottom of the description. Locations are TBD, but will be updated at the above links as soon as they’re finalized.

A further series is planned for several US cities in March, and we’ll be sure to announce them as well.

This post is a little geeky and technical and product-related for AgileAdvice, and is a shameless self-promotion. Nevertheless, since testability, test-driven-development, and incremental design are non-exclusive sub-topics of Agile, I though I’d report this here anyway.

Many developers use the Dependency Injection and Inversion of Control (IoC) patterns through such IoC containers as Spring, Hivemind, Picocontainer, and others. They have all sorts of benefits to testability, flexibility, etc. that I won’t repeat here, but can be read about here, here, and here. A great summary of the history of “IoC” can be found here. J2ME developers, however, especially those on limited devices that use the CLDC configuration of J2ME, cannot use the substantial number of IoC/DI containers out there, because they nearly all rely on reflection. These also often make use of APIs not present in the CLDC – APIs which could not easily be added. Lastly there’s a tendency among developers of “embedded software” to be very suspicious of complexity.

In working out some examples of DI as part of a testability workshop at one of my clients, I whipped up a quick DI container, and being the freak that I am, hardened it until it was suitable for production, because I hate half-finished products. So allow me to introduce the Israfil Micro Container. (That is, the Container from the Israfil Micro project). As I mention in the docs, “FemtoContainer” just was too ridiculous, and this container is smaller than pico-container. The project is BSD licensed, and hosted on googlecode, so source is freely available and there’s an issue/feature tracker, yadda yadda.

Essentially I believe that people working on cellphones and set-top boxes shouldn’t be constrained out of some basic software design approaches – you just have to bend the design approach to fit the environment. So hopefully this is of use to more than one of my clients. It currently supports an auto-wiring registration, delayed object creation (until first need), and forthcoming are some basic lifecycle support, and a few other nicities. It does not use reflection (you use a little adapter for object creation instead), and performs quicker than pico-container. Low, low overhead. It’s also less than 10 classes and interfaces (including the two classes in the util project). It’s built with Maven2, so you can use it in any Maven2-built project with ease, but of course you can always also just download the jar (and the required util jar too). Enjoy…

P.S. There are a few other bits on googlecode that I’m working on in the micro-zone. Some minimalist backports of some of java.lang.concurrency (just the locks), as well as some of the java.util.Collections stuff. Not finished, but also part of the googlecode project.

This was actually cribbed from a Bruce Schneier blog post about security…

Security engineers see the world differently than other engineers. Instead of focusing on how systems work, they focus on how systems fail, how they can be made to fail, and how to prevent–or protect against–those failures. Most software vulnerabilities don’t ever appear in normal operations, only when an attacker deliberately exploits them. So security engineers need to think like attackers.People without the mindset sometimes think they can design security products, but they can’t. And you see the results all over society–in snake-oil cryptography, software, Internet protocols, voting machines, and fare card and other payment systems. Many of these systems had someone in charge of “security” on their teams, but it wasn’t someone who thought like an attacker.  

There’s an interesting parallel between this statement and how most software quality is handled. Quality and Security are similar. In fact, I see security as a very specific subset of quality-mindedness. Certainly both require the same mindset to ensure – rather than thinking merely “how will this work”, a quality-focused person will also, or perhaps alternately think: “how might this be breakable”. From this simple change in thinking flows several important approaches

• Constraint-based thinking (as opposed to solution based thinking): allows an architect/developer to conceive of the set of possible solutions, rather than an enumeration of solutions. By looking at constraints, a developer implements the lean principle of deciding as late as possible, with as full information as possible.
• Test-First: As one thinks of how it might break, scenarios emerge that can form the basis of test cases. These cases form a sort of executable acceptance criteria
• Lateral Thinking: The constraint+test approach starts to get people into a very different mode, where vastly different kinds of solutions show up. The creative exercise of trying to break something provides insights that can change the whole approach of the system.

 Schneier goes on to ponder 

This mindset is difficult to teach, and may be something you’re born with or not. But in order to train people possessing the mindset, they need to search for and find security vulnerabilities–again and again and again. And this is true regardless of the domain. Good cryptographers discover vulnerabilities in others’ algorithms and protocols. Good software security experts find vulnerabilities in others’ code. Good airport security designers figure out new ways to subvert airport security. And so on.  

 Here again – I think it’s possible to help people get a mind-set about quality, but some do seem to have a knack. It’s important to have some of these people on your teams, as they’ll disturb the waters and identify potential failure modes. These are going to be the ones who want to “mistake proof” (to borrow Toyota’s phrase) the system by writing more unit tests and other executable proofs of the system. But most importantly (and I can personally testify to this) it is critical that people just write more tests. It is a learned skill to start to think of “how might this fail” until it becomes a background mental thread, always popping up risk models.A related concept is Demmings’ “systems-thinking”, which, applied to software quality, causes one to start looking at whole ecosystems of error states. This is when fearless re-factoring starts to pay off, because the elimination of duplication allows one to catch classes of error in fewer and fewer locations, where they’re easier to fix. There are many and multifarious spin-off effects of this inverted questioning and the mindset it generates. Try it yourself. When you’re writing code, ask yourself how you might break it? What inputs, external state, etc. might cause it to fail, crash, or behave in odd ways. This starts to show you where you might have state leaking into the wild, or side-effects from excessively complex interactions in your code. So quality focus can start to improve not only the external perception of your product, but also its fitness to new requirements by making it more resilient and less brittle. Cleaner interactions and less duplication allow for much faster implementation of new features.I could go on, but I just wanted to convey this sense of “attitude” or “mindset,” over mere technique. Technique can help you get to a certain level, but you have to let it “click”, and the powerful questions can sometimes help.

Building software is expensive. I’m not talking about creating software, I mean taking software as written (source code) and running it through compilers and linkers and post-processors and packagers and obfuscators and installer-generators. It might not seem so, but look under the covers and you will find a wealth of costs and potential savings…

Lifecycle of the Developer

The developer has a concept he needs to translate into software. He (or she) does not sit and meditate until it comes to him, then streams it effortlessly into the computer. Rather he tries something, and tries something else, and writes some conditions (tests) to limit the scope of his options, and cycles over and over and over again between four main activities: creating -> building -> executing tests -> discovering. The developer then wraps around, having discovered and learned (found the bug or identified a future direction) and begins to create again.

If you break this down, there are two states – active and waiting – that the developer is in at any point. He is active when he is learning and he is active when he is creating. He is waiting when he is building and executing tests. So the developer’s ability to do further learning/creative work comes from how long he has to wait for building/executing the software.

None of the above would be interesting except for the fact that the developer does so many times per day. How many times? Well, now we get to the math.

Large Builds = Fewer Builds

Most software I’ve seen in my career has been heavy, bloated software. Lots of deep interlinking of code with lots of tests that go from the front of the system to the back, etc. On a typical build machine, this has often translated into 5-20 minutes to build and execute a subset of the automated tests.

When the build is this large, developers will tend to optimize their behaviour around this limitation. If I, as a developer, have to wait for 5-20 minutes between changes, I will start to group up changes into larger and larger batches so the total cost of building as a percentage of my time goes down. This may mean that I work on several things and then rebuild. It can be easy to lose track of all the balls being juggled, but worse – this behaviour encourages a reduction in test execution. I might also decide that I’m going to only build and not execute my tests, because they are taking, say, 75% of the total time waiting.

More changes + fewer tests = larger difficulties if the code is not exactly perfect or if I didn’t think of everything.

Each time I build, then, I risk breaking more, as any small change I introduce will be harder to find, and possibly have more effects. And I’m less likely to see the problem for many of these cycles due to the loss of the feedback from the tests. And the longer an error persists, the more likely it will compound with other errors, etc.

Reduced Value from Experienced Developers

This is not an absolute, but what I have found is that the longer the build, the less you are likely to obtain as much relative value from an experienced developer, vs. his less experienced counterparts. This is simply because less of the day is spent doing the tasks of development – creating and learning, and more time is spent waiting. Any monkey can wait, but high-value developers are high-cost and making them wait reduces the value you get from that increased developer productivity.

“Hold on a minute, Bub,” you might say, “an experienced developer will not stand for that delay, and will take steps to improve his conditions…”

It sort of makes the point. The more experienced the developer, the less likely he is to stomach the delay, as he feels the tangible loss of his value to the development effort. Often such developers will either batch work as discussed above, with all its issues, or will work to shorten build times.

Shortening Build Times – The path to developer productivity

The shorter you can make build times, the greater percentage of time can be spent in creative and learning tasks, without the accompanying batching problems. The shorter you can make builds, the more frequently you can execute tests. The more you shorten builds, the less waste in your developer’s lifecycle.

But how?

Spend the money on better hardware

I’m not kidding. Paying $3000/developer for better equipment (just to take a number out of the air)… yes, for EACH developer, is a remarkably small cost when you consider the whole picture. If a developer makes $50,000/year (doesn’t matter if that’s high or low, it’s simple math… adjust for your market) and if they’re being slowed down by 20% by their builds being 10 minute builds (again, conservative slowdown for that high a build length), then consider: If they’re slowed down by 20%, then you’re throwing away $10,000/year on wasted developer productivity. Even more for higher-paid developers. Hardware is almost always cheaper than developer time, and if they’re using equipment that they complain is too slow, then it’s a very small investment for a large payoff.

Break the projects into smaller sub-projects

If you have a large project and containing, say, 6000 classes, 21,000 lines of code, and lots of interdependencies, plus tests, that’s a hefty build. But if I change one class in the middle, not every last thing has to be completely rebuilt. Yes javac (if you use it) or other change-sensitive compilers or build systems can do some speedup on that, but you still end up running all the tests, because you don’t know inherently which tests are not relevant because of the dependencies.

If you break the projects into smaller sub-projects, and then use some central dependency management system such as Maven, and a continuous integration system that understands the dependencies, it can build those projects with changes, and then only build the down-stream dependencies

This has many side-benefits. For one, smaller sub-projects means fewer circular dependencies, which improves reusability of those units. Further more, smaller units are easier to test, and developers are less intimidated when writing tests for smaller units. This “virtuous cycle” means developers are writing more tests, which improves quality which wastes less developer time on later defects, etc.

Using such systems can also provide more isolation between these units which reduces contention when many developers are working on a system simultaneously, as there are fewer places where developers would have to integrate their changes directly with another’s, and such locations become easier to track and manage.

Encourage more unit testing

This is tricky, because it’s easy for a developer to think that writing a single set of system tests which tests all the way through the system is faster than writing a larger volume of isolated unit tests. Below the surface, you begin to find that by executing a long (1-2 minute) integration test, and perhaps executing 5-15 of them can very quickly result in 5-30 minute builds. These end up then either getting run and delaying development, or they don’t get run at all by the developer until much later. This latter case, alluded to above is the bane of software quality, because it means that large volumes of change are being introduced without any sort of regression test to ensure that error hasn’t been introduced. If integration tests are run once per day (and you don’t have enough isolated unit tests to cover the same code) then you can have a day’s worth of several developers’ error colliding, which can cause hours of work teasing apart the problems.

Unit tests should execute in under 1 second, in my experience (perhaps 2-5 seconds for tests that involve multithreading). You should be able to have hundreds of unit tests, and a good test execution framework should execute many of them in parallel to speed up test execution. (Maven’s surefire doesn’t do this per-se, but does support TestNG which certainly supports parallel execution of tests). Furthermore, isolated unit testing works with hard-coded test data which cannot be changed by this or that environment change caused by other teams or infrastructure groups. They are portable tests. They should work in any environment, with rare exceptions.

Additionally, isolated testing encourages a stronger “code-to-contract” or “strong API” approach to design, which has long been understood to improve quality. Such approaches mean that you can have more complete code and branch coverage, but probably more importantly, more case or scenario coverage.

The Bottom Line

Ultimately, software is about delivering as much value as possible, sustainably, as early as possible. This is true in both commercial and open-source contexts, but is certainly and especially true in a commercial context. Every delay in developer productivity is a delay in realizing the value provided, which often equates to a delay in gaining revenue. By reducing the wait-time for developers due to build and test execution you can achieve at least large fractional gains, and sometimes you achieve economies that allow an order of magnitude in productivity gains from the combined factors mentioned above. If you start to get the feeling that “it costs too much”, or “writing all those tests takes time,” or “re-organizing the projects will be too disruptive,” make sure you’re thinking of the whole cost. Don’t sub-optimize on a part of the value at the cost of delivering value to your customer, sustainably, as early as possible. Choosing any path that slows this value stream down is throwing money away, even if it seems to save costs/time in a localized way. “Penny-wise, Pound(Dollar)-foolish” has never been more true.

There is an interesting article called “Software Estimates and the Parable of the Cave“. It starts out well. The cave parable is effective and clearly conveys the problem with estimating software work. However, there is a big section of the article called “Applying this Wisdom” which I think does a dis-service to everyone. Let’s look at this closely…

“Draw on your experience.” Okay, didn’t we just get through a parable that said that experience was inapplicable, or at best questionable? Okay, let’s take that as a given. So what can we do? Don’t estimate. Seriously.

“Expect the unexpected!” I love this one. This is like saying tell me what you don’t know that you don’t know! Or please tell me the price (on Dec. 20th, 2009) of the shares of the 20th (after today) startup to go public on the NASDAQ. The future is unpredictable. The farther out, the worse it is. So what can we do? Don’t estimate.

“Provide estimates that are ranges…” I’m not sure why anyone would want an estimate that is a range. Are you going to give me a money-back guarantee that you will deliver at the end of the range? Are you saying that if I through all the money and resources and intelligence at you you will be able to deliver at the low end of your range? Are you saying that you don’t know? So what can we do…(wait for it!)? … Don’t estimate.

“Make your estimates as fine-grained as possible”. Let’s take this one to the limit shall we? The finest grained estimate we can make is based on complete knowledge of every single tiny task needed in order to create the software… which is logically equivalent to writing the software and then telling me how long it actually took. Yes, this is getting a bit repetitive… so what can we do… (let’s hear it from the audience!!!). Don’t estimate.

“Use incremental and iterative processes”. Okay, I admit, I _like_ this one. But how does it relate to estimates exactly? Not exactly clear, but I suppose I have to admit, I have a way to do this in mind… if you must. (But it has absolutely nothing to do with what is recommended in “applying the wisdom” above. Oh… and Don’t estimate.

“Work on the riskiest part of the system first”. Jeez Louise! Let’s pretend I’m running a business. I tell you: “here’s all the stuff I need and want”. You tell me: “this part ‘A’ is the riskiest, I’m going to work on that first.” I say, “screw you! I want the most important thing first!” And you say…. what exactly? (Well, if the most important is also the riskiest, then actually I say “okay”.) And what does this have to do with estimating?

“Environment of trust.” I like this one too. But I don’t think it has anything to do with estimation except as a surface symptom (failure to meet estimates should _not_ lead to more work trying to get better at estimating). I think it has to do with truthfulness, quality and a few other things beyond the scope of this article…

In agile software development, automated testing plays a big role due to the emphasis on quality. A friend of mine, Tom Cooper, whom I worked with a few years ago at a major capital markets firm, passed these links on to me yesterday. There are some folks at NIST and the Univ. of Texas who are building a tool called FireEye. Here is the press release, here is the pdf presentation.

Lisa Crispin has written an excellent article about defect tracking in agile environments. The article is a couple of months old now, but if you haven’t read it yet, you definitely should! I particularly like the perspective that Mary Poppendieck offers in the article…

Since Quality is Not Negotiable, and since defects in both XP and Scrum are meant to be dealt with immediately, there is a good argument to be made for doing away with any type of automated tool-based defect tracking.

One team I coached really struggled with this. They were working on the first every agile project in their organization and none of the team members had any prior experience with agile or with test-driven development. It was a small project, but management wanted to do things “right”. They had a team room, a cross-functional team, short iterations, etc. etc.

One member of the team was a person who was their Quality Assurance expert. This person was extremely capable: able to write excellent test scripts (in Excel) and able to think carefully about quality. However, the organization was not set up to support an agile approach to quality: QA folks were measured on how many defects they found and logged in their defect tracking system.

Since the team had decided to use test-driven development, they soon found that they had an odd question: when does incomplete work become defective work that needs to be logged? In other words, since the tests are written first and they start out by failing, and since this is normal and happens for every single feature, exception condition and even method of code, tracking all these as “defects” would be meaningless. So how do you decide that something is a defect instead of test-driven work that just isn’t yet making the tests pass?

After a great deal of discussion and confusion and consternation, we finally agreed to a simple and effective practice: inside of an iteration, any issues or problems that come up are written on the whiteboard. The whiteboard needs to be clear by the end of the iteration. If it is not, then the items that are still there are logged into the defect tracking system. These items also become top priority for the next iteration. As well, any items which had non-passing tests by the end of the iteration were also put into the defect tracking system and became top priority for the next iteration.

This helped… and we only ended up with at most one or two items at the end of each iteration that could be logged… and often there were no known defects.

This was legitimately painful for the QA person whose defect logging rates went way down. Fortunately, since this was a pilot project, it was easy to explain to management what was going on. This is a very good example of how a management practice that makes a lot of sense in a non-agile environment becomes a huge obstacle in an agile environment.

Is boosting productivity simple? Yes! Just cut code! One problem here. The gap between code complete and feature done is bigger then it appears. That was my thought as I was looking at statistics.

Reposted [and edited - Mishkin] from www.softwarefrontier.com by Dmitri Zimin(e)

On average, a developer cuts 300 lines per day, some true heroes claim more. This “productivity” comes with the price of 100 defects per 1000 lines of code. Finding defect and fixing it is on average 4 to 16 hours [1] [2]

Do the math: Your star developer produces 500 lines a day. This creates up to 80 hours of extra work to himself, the dev team and their tester buddies. 10 days! I call it “grey scope creep”.

At first, these statistics seems totally off. But you can surely remember such a super-star code monkey? You surely know a few. Now go over all these steps to fix a defect: cutting the build, installing it in QA, testing, finding the bug, writing a bug report, assigning it back to development, reviewing, trying in vain to reproduce… “Works on my machine”, sending it back to QA, and finally, after a couple of round-trips, it’s fixed. Then regression sticks out its ugly head, and the high 16 man-hours per bug looks too good to be true.

So, is boosting productivity simple? Yes! Less bugs in, more bugs out! One problem here. But enough on problems, let’s get positive and move on to how fight grey scope creep.

First, if these superstar developers only care about the joy of cutting code, not shipping the software, they get the boot. Don’t regret it, they produce more work then they get done. It is not about developers developing and testers testing. It is developers and testers working cooperatively to ship software. We’ve got no space for local optimization.

Second, good software practices are to the rescue. Pair programming, design and code reviews, test driven development, etc… They surely slow down the LOC/day pseudo-performance (less bugs in) and reduce the number of defects. Pair programming by itself brings 15% less bugs, with healthy 15% slow-down of LOC performance [3].

Third, redefine “done”. Fresh features are marked done, and then disappear somewhere in QA to eventually fire back at unknown time with unknown bugs. Grey scope creep. Stop it. Instead, insist on taking less but making it “done” within an iteration. Done reads fully developed, thoroughly tested, debugged, fixed including regression, and accepted by product owner. If this brings development to a deadlock, it’s one of the two. You may be blessed to fail faster and cheaper. After all, if you can’t get “done” just one feature, how would you ever be done with the entire project? Or the team overcomes the crisis and changes the working habits. The shortened feedback loop and fixing bugs as you make it leads to less time per bug fix. The grey scope doesn’t creep beyond the iteration. Problems become obvious early enough to to deal with them.

Finally, until you fix the grey scope creep, your estimations could be off, up to 10 times! Measure well.

So, is boosting productivity simple? Yes! One problem here. Trying to address it may expose the stakeholders to too much truth about the state of the project. Figure out how to deal with this truth. Or wait to my next blog entry… I have some thoughts about this too.

Last night I was thinking more about the analogy of technical debt. In this analogy, design and quality flaws in a team’s work become a “debt” that must eventually be paid back. This analogy is fantastic because it can be taken just a little bit further to understand just how bad defects are…

Debt is sometimes useful. Financial debt can be used for risk reduction, investment, and emergencies. But it can also cause problems. Too much debt becomes hard to manage. If debt maintenance costs more than revenue (minus other expenses), then you’re going down!

Technical debt is a little different than financial debt.

Suppose I went into a boardroom full of high-power executives for some large company. (Never mind how I got there.) And I pitched this fabulous idea: I’ll give them a bunch of money for them to use for operations, expansion, whatever. All they have to agree to is that I choose the interest rate when I ask for repayment… trust me!

I would be thrown out of that room so fast!

That is what we do when we encourage teams to take on technical debt.

There is no way to know the interest rate on a defect. Part of the cost of a defect is obvious: how much time and materials will it take to repair the immediate problem. (Although even that is often hard to measure.) But there are also lots of non-obvious and probably non-measurable costs. How much effort will it take to get to the root cause of the defect so that it doesn’t occur a second time? How much will it affect our “goodwill” and thus reduce further and repeat sales? How much will the existence of one defect hide the existence of other defects (with their own costs)? How much will the defect demoralize the team and increase staff turnover or reduce productivity? How much of an opportunity will the defect create for competitors? How much will the defect increase maintenance and support costs?

In other words, every time someone asks a team to let quality slide, they are asking the team (and the organization) to take on debt with an unknown interest rate. Which is lunacy.

Read more about this:

Quality is Not Negotiable

Voluntary Technical Debt – James Shore talks about a situation where a conscious decision to take on technical debt led to positive results.

Technical Debt: The Threshold of Acceptable Pain – Steve Bate talks about skill and sensitivity to technical debt. Here’s a great quote from this article:

What if someone has a very high threshold of pain? I’d expect to see lots of crud (technical debt) in their code…. The high threshold developer doesn’t mind spending weeks on new features that would otherwise require days or hours with clean code. And they don’t mind staying at work all night fixing bugs before a major release or spending countless hours babysitting the production system after it’s been released. It doesn’t seem to bother them to spend significant time away from family and friends or to have limited time or other interests and hobbies. …they sometimes experience pleasure at being the “hero” who saved the company from the bugs they created.

An Incremental Technique to Pay Off Testing Technical Debt – Johanna Rothman talks about technical debt and describes a simple risk-oriented approach to reducing it.