On Agile, TDD, and other Software Engineering processes

I've been looking around the job market recently, and one thing that's become much more visible since I last did so is the requirement to have experience in some form of "Agile" software engineering process. While I'm not going to argue that the requirement is pointless, I do wonder how much of it is from companies who don't understand what they really want.

To understand where I'm coming from, you must first understand what an engineering process actually is. As a competent engineer, you develop patterns of work that help you produce good results every time. An engineering process is what you have when you look at a pattern of work, and write it down with explanations of why you do something. For example, test-driven development (TDD) is based on the observation that, when working with a pre-existing system, the safest way to improve things is to write tests for the functionality that you want. As long as those tests keep passing, you know the system works; as soon as they fail, you know that you've broken something important. TDD takes that further - you always write tests first for everything, even new features. You know that the new tests will fail, and you write just enough code to get the new tests passing; because you've got full test coverage, you can safely engage in quite major rework of the code, knowing that the test suite will alert you to bugs.

Another technique is pair programming. This relies on the observation that it's easier to spot other people making mistakes than it is to stop yourself making similar mistakes. So, put two programmers at one computer; have one of them write code, the other chiming in with observations and questions. Swap roles frequently, so that neither programmer gets bored.

The commonality here is that a good engineer will use whichever process is appropriate for the work they're doing; if I'm enhancing an existing feature, I will do TDD, as I can't risk breaking the existing feature, and it's easier to produce a better feature rather than a mudball of related features if I'm having to think about how it will be used as I write a test case. Similarly, if I'm deep in a complicated problem, I'll pair-program with another programmer who's as competent as I am; they force me to explain things that I think are obvious, and, as it turns out, sometimes what I've assumed is obvious is not only not obvious, but is also key to solving the problem.

It should at this point be obvious why I think that asking for experience of Agile processes isn't necessarily going to get companies what they're after. An Agile process isn't in itself helpful - they provide nice shorthands for good engineering processes, but you can practice Agile processes for years without ever doing a decent bit of engineering. For example, if you're doing TDD, you can write useless test cases, or write code that does a little bit more than the area the test case is supposed to cover. Before you know it, you're claiming to do TDD, but you're actually back in big ball of mud disaster programming territory.

Similarly, pair programming works because the members of the pair are of similar levels of competence - if you pair up with programmers who are considerably worse than you, you don't benefit from it. However, it's possible to do mentoring (pair a good programmer and a bad one together, so that the bad one can learn from the good one), and call it pair programming; from the outside, they look the same.

Practically, you don't normally want to stick with one Agile process slavishly; the core principles underlying all Agile processes are good (short release cycles, don't try and predict when you can adapt during the development process, don't waste time on something you might never need), but which detailed process is best depends strongly on what you're doing today. A good engineer knows enough to insist on pair programming when they're in a complicated problem, TDD when they're trying not to break an existing feature, Scrum-style sprints when appropriate, whatever parts of Agile will improve things today. A bad engineer will do TDD badly, so that it doesn't show gains, will use Scrum-style sprints to avoid facing up to the hard problems, pair programming as an excuse to goof off, and will generally "have experience in Agile processes", yet not show any gain from them.

What employers should really be looking for is good, adaptable engineers; these people will adjust to whatever process you have in place, will look to change it where it doesn't work, and won't hide behind buzzwords. Asking for "Agile processes" is no longer a way to catch engineers who keep up with the profession - it's now a way of catching people who know what the buzzwords are.

Having said that, I don't know what today's version of "Agile processes" should be; you need something that's new on the scene, that good engineers will be exposing themselves to and learning about, and that isn't yet well-known enough to encourage bad engineers to try and buzzword bingo their way past the HR stage.