- retiring Oracle data warehouse licenses / servers,
- retiring disk space / hardware,
- and saving CPU time with new hardware
Pair Programming Economics by Olaf Lewitz describes three activities in programming: typing, problem-solving and reading code. How does pair programming help? By making the balance between those three activities better.
I was poling around for a good definition of DevOps and found a thoughtful article written by Ernest Mueller called What is DevOps? Highly recommended reading as it includes lots of insight about the relationship between Agile and DevOps. FWIW, I feel that the concept of the Definition of “Done” is Scrum’s own original take on the same class of ideas: breaking down silos in an organization to get stuff into the marketplace faster and faster. I even talked about operationalizing software development back in 2004 and 2005 as a counterpoint to the project management approach which puts everyone in silos and pushes work through phase gates.
Whenever I run a Certified Scrum Product Owner training session, one concept stands out as critical for participants: the relationship of the Product Owner to the technical demands of the work being done by the Scrum team.
The Product Owner is responsible for prioritizing the Product Backlog. This responsibility is, of course, also matched by their authority to do so. When the Product Owner collaborates with the team in the process of prioritization, there may be ways which the team “pushes back”. There are two possible reasons for push-back. One is good, one is bad.
Bad Technical Push-Back
The team may look at a product backlog item or a user story and say “O gosh! There’s a lot there to think about! We have to build this fully-architected infrastructure before we can implement that story.” This is old waterfall thinking. It is bad. The team should always be thinking (and doing) YAGNI and KISS. Technical challenges should be solved in the simplest responsible way. Features should be implemented with the simplest technical solution that actually works.
As a Product Owner, one technique that you can use to help teams with this is that when the team asks questions, that you aggressively keep the user story as simple as possible. The questions that are asked may lead to the creation of new stories, or splitting the existing story. Here is an example…
Suppose the story is “As a job seeker I can post my resume to the web site…” If the technical team makes certain assumptions, they may create a complex system that allows resumes to be uploaded in multiple formats with automatic keyword extraction, and even beyond that, they may anticipate that the code needs to be ready for edge cases like WordPerfect format. The technical team might also assume that the system needs a database schema that includes users, login credentials, one-to-many relationships with resumes, detailed structures about jobs, organizations, positions, dates, educational institutions, etc. The team might insist that creating a login screen in the UI is an essential prerequisite to allowing a user to upload their resume. And as for business logic, thy might decide that in order to implement all this, they need some sort of standard intermediate XML format that all resumes will be translated into so that searching features are easier to implement in the future.
It’s all CRAP, bloat and gold-plating.
Because that’s not what the Product Owner asked for. The thing that’s really difficult for a team of techies to get with Scrum is that software is to be built incrementally. The very first feature built is built in the simplest responsible way without assuming anything about future features. In other words, build it like it is the last feature you will build, not the first. In the Agile Manifesto this is described as:
Simplicity, the art of maximizing the amount of work not done, is essential.
The second feature the team builds should only add exactly what the Product Owner asks for. Again, as if it was going to be the last feature built. Every single feature (User Story / Product Backlog Item) is treated the same way. Whenever the team starts to anticipate the business in any of these three ways, the team is wrong:
Okay, but what about architecture? Fire your architects. No kidding.¹
Good Technical Push-Back
Sometimes stuff gets non-simple: complicated, messy, hard to understand, hard to change. This happens despite us techies all being super-smart. Sometimes, in order to implement a new feature, we have to clean up what is already there. The Product Owner might ask the Scrum Team to build this Product Backlog Item next and the team says something like: “yes, but it will take twice as long as we initially estimated, because we have to clean things up.” This can be greatly disappointing for the Product Owner. But, this is actually the kind of push-back a Product Owner wants. Why? In order to avoid destroying your business! (Yup, that serious.)
This is called “Refactoring” at it is one of the critical Agile Engineering practices. Martin Fowler wrote a great book about this about 15 years ago. Refactoring is, simply, improving the design of your system without changing it’s business behaviour. A simple example is changing a set of 3 radio buttons in the UI to a drop-down box with 3 options… so that later, the Product Owner can add 27 more options. Refactoring at the level of code is often described as removing duplication. But some types of refactoring are large: replacing a relational database with a NoSQL database, moving from Java to Python for a significant component of your system, doing a full UX re-design on your web application. All of these are changes to the technical attributes of your system that are driven by an immediate need to add a new feature (or feature set) that is not supported by the current technology.
The Product Owner has asked for a new feature, now, and the team has decided that in order to build it, the existing system needs refactoring. To be clear: the team is not anticipating that the Product Owner wants some feature in the future; it’s the very next feature that the team needs to build.
This all relates to another two principles from the Agile Manifesto:
Continuous attention to technical excellence and good design enhances agility.
The best architectures, requirements, and designs emerge from self-organizing teams.
In this case, the responsibilities of the team for technical excellence and creating the best system possible override the short-term (and short-sighted) desire of the business to trade off quality in order to get speed. That trade-off always bites you in the end! Why? Because of the cost of fixing quality problems increases exponentially as time passes from when they were introduced.
Refactoring is not a bad word.
Keep your code clean.
Let your team keep its code clean.
Oh. And fire your architects.
¹ I used to be a senior architect reporting directly to the CTO of Charles Schwab. Effectively, I fired myself and launched an incredibly successful enterprise architecture re-write project… with no up-front architecture plan. Really… fire your architects. Everything they do is pure waste and overhead. Someday I’ll write that article
From the excellent Ron Jeffries, we have “Refactoring – Not on the Backlog!“
I am regularly asked for examples of where Agile Practices could be used that are not related to software development.
I recently came across this video and just had to share the link to it.
The company encourages all team members to participate, keeps things time-boxed and makes appropriate use of subject matter experts.
It’s awesome to see what can be done in 5 days.
I’ve just submitted a proposed session to the Agile 2010 web site:
Please go to the site and let me know in the comments there if you have any suggestions about the proposal!
Isráfíl Consulting is finally prepared for the first series of its Agile Software Engineering Practices training courses. This series is offered in partnership with Berteig Consulting who are graciously hosting the registration process. Their team has also helped greatly in shaping the presentation style and structure of the course. The initial run will be in Ottawa, Toronto (Markham), and Kitchener/Waterloo.
Topics covered will include Test Driven Development (TDD), testability, supportive infrastructure such as build and continuous integration, team metrics, incremental design and evolutionary architecture, dependency injection, and so much more. (This course won’t present the planning side of XP, but covers many other aspects common to XP projects) It makes a great complement for training in Agile Processes such as XP, Scrum, or OpenAgile. The overview slide presentation is available for free download from the Isráfíl web site.
The courses are scheduled for:
The course is $1250 CAD per student, and participants receive a transferrable discount of $100 CAD for other training with Berteig Consulting as a part of our ongoing partnership. I initially prototyped this course in Ottawa this December, and am very excited to see this through in several locales. Class size is limited to 15, so we can keep the instruction style more involved. The above schedules are linked to Berteig Consulting’s course system and have registration links at the bottom of the description. Locations are TBD, but will be updated at the above links as soon as they’re finalized.
A further series is planned for several US cities in March, and we’ll be sure to announce them as well.
This post is a little geeky and technical and product-related for AgileAdvice, and is a shameless self-promotion. Nevertheless, since testability, test-driven-development, and incremental design are non-exclusive sub-topics of Agile, I though I’d report this here anyway.
Many developers use the Dependency Injection and Inversion of Control (IoC) patterns through such IoC containers as Spring, Hivemind, Picocontainer, and others. They have all sorts of benefits to testability, flexibility, etc. that I won’t repeat here, but can be read about here, here, and here. A great summary of the history of “IoC” can be found here. J2ME developers, however, especially those on limited devices that use the CLDC configuration of J2ME, cannot use the substantial number of IoC/DI containers out there, because they nearly all rely on reflection. These also often make use of APIs not present in the CLDC – APIs which could not easily be added. Lastly there’s a tendency among developers of “embedded software” to be very suspicious of complexity.
In working out some examples of DI as part of a testability workshop at one of my clients, I whipped up a quick DI container, and being the freak that I am, hardened it until it was suitable for production, because I hate half-finished products. So allow me to introduce the Israfil Micro Container. (That is, the Container from the Israfil Micro project). As I mention in the docs, “FemtoContainer” just was too ridiculous, and this container is smaller than pico-container. The project is BSD licensed, and hosted on googlecode, so source is freely available and there’s an issue/feature tracker, yadda yadda.
Essentially I believe that people working on cellphones and set-top boxes shouldn’t be constrained out of some basic software design approaches – you just have to bend the design approach to fit the environment. So hopefully this is of use to more than one of my clients. It currently supports an auto-wiring registration, delayed object creation (until first need), and forthcoming are some basic lifecycle support, and a few other nicities. It does not use reflection (you use a little adapter for object creation instead), and performs quicker than pico-container. Low, low overhead. It’s also less than 10 classes and interfaces (including the two classes in the util project). It’s built with Maven2, so you can use it in any Maven2-built project with ease, but of course you can always also just download the jar (and the required util jar too). Enjoy…
P.S. There are a few other bits on googlecode that I’m working on in the micro-zone. Some minimalist backports of some of java.lang.concurrency (just the locks), as well as some of the java.util.Collections stuff. Not finished, but also part of the googlecode project.
Security engineers see the world differently than other engineers. Instead of focusing on how systems work, they focus on how systems fail, how they can be made to fail, and how to prevent–or protect against–those failures. Most software vulnerabilities don’t ever appear in normal operations, only when an attacker deliberately exploits them. So security engineers need to think like attackers.People without the mindset sometimes think they can design security products, but they can’t. And you see the results all over society–in snake-oil cryptography, software, Internet protocols, voting machines, and fare card and other payment systems. Many of these systems had someone in charge of “security” on their teams, but it wasn’t someone who thought like an attacker.Â Â
There’s an interesting parallel between this statement and how most software quality is handled. Quality and Security are similar. In fact, I see security as a very specific subset of quality-mindedness. Certainly both require the same mindset to ensure – rather than thinking merely “how will this work”, a quality-focused person will also, or perhaps alternately think: “how might this be breakable”. From this simple change in thinking flows several important approaches
Â Schneier goes on to ponderÂ
This mindset is difficult to teach, and may be something you’re born with or not. But in order to train people possessing the mindset, they need to search for and find security vulnerabilities–again and again and again. And this is true regardless of the domain. Good cryptographers discover vulnerabilities in others’ algorithms and protocols. Good software security experts find vulnerabilities in others’ code. Good airport security designers figure out new ways to subvert airport security. And so on.Â Â
Â Here again – I think it’s possible to help people get a mind-set about quality, but some do seem to have a knack. It’s important to have some of these people on your teams, as they’ll disturb the waters and identify potential failure modes. These are going to be the ones who want to “mistake proof” (to borrow Toyota’s phrase) the system by writing more unit tests and other executable proofs of the system. But most importantly (and I can personally testify to this) it is critical that people just write more tests. It is a learned skill to start to think of “how might this fail” until it becomes a background mental thread, always popping up risk models.A related concept is Demmings’ “systems-thinking”, which, applied to software quality, causes one to start looking at whole ecosystems of error states. This is when fearless re-factoring starts to pay off, because the elimination of duplication allows one to catch classes of error in fewer and fewer locations, where they’re easier to fix. There are many and multifarious spin-off effects of this inverted questioning and the mindset it generates. Try it yourself. When you’re writing code, ask yourself how you might break it? What inputs, external state, etc. might cause it to fail, crash, or behave in odd ways. This starts to show you where you might have state leaking into the wild, or side-effects from excessively complex interactions in your code. So quality focus can start to improve not only the external perception of your product, but also its fitness to new requirements by making it more resilient and less brittle. Cleaner interactions and less duplication allow for much faster implementation of new features.I could go on, but I just wanted to convey this sense of “attitude” or “mindset,” over mere technique. Technique can help you get to a certain level, but you have to let it “click”, and the powerful questions can sometimes help.
Building software is expensive. I’m not talking about creating software, I mean taking software as written (source code) and running it through compilers and linkers and post-processors and packagers and obfuscators and installer-generators. It might not seem so, but look under the covers and you will find a wealth of costs and potential savings…
The developer has a concept he needs to translate into software. He (or she) does not sit and meditate until it comes to him, then streams it effortlessly into the computer. Rather he tries something, and tries something else, and writes some conditions (tests) to limit the scope of his options, and cycles over and over and over again between four main activities: creating -> building -> executing tests -> discovering. The developer then wraps around, having discovered and learned (found the bug or identified a future direction) and begins to create again.
If you break this down, there are two states – active and waiting – that the developer is in at any point. He is active when he is learning and he is active when he is creating. He is waiting when he is building and executing tests. So the developer’s ability to do further learning/creative work comes from how long he has to wait for building/executing the software.
There is an interesting article called “Software Estimates and the Parable of the Cave“. It starts out well. The cave parable is effective and clearly conveys the problem with estimating software work. However, there is a big section of the article called “Applying this Wisdom” which I think does a dis-service to everyone. Let’s look at this closely…
In agile software development, automated testing plays a big role due to the emphasis on quality. A friend of mine, Tom Cooper, whom I worked with a few years ago at a major capital markets firm, passed these links on to me yesterday. There are some folks at NIST and the Univ. of Texas who are building a tool called FireEye. Here is the press release, here is the pdf presentation.
Lisa Crispin has written an excellent article about defect tracking in agile environments. The article is a couple of months old now, but if you haven’t read it yet, you definitely should! I particularly like the perspective that Mary Poppendieck offers in the article…
Is boosting productivity simple? Yes! Just cut code! One problem here. The gap between code complete and feature done is bigger then it appears. That was my thought as I was looking at statistics.