When Should We Do TDD?

On a recent episode (78) of This Agile Life, my fellow hosts talked about when to do Test-Driven Development (TDD). They all said that you should always do TDD — at least for anything that will go into production; there’s an exception for experimenting or “spiking”.

I wasn’t on that episode, but later commented on the topic. (Episode 83 — which wasn’t really all that great.) My take was slightly different. I said that you should do TDD only when the benefits outweigh the costs. Unfortunately, we usually greatly underestimate the benefits. And the costs often seem high at first, because it takes some time to get good at writing automated tests. Not to mention that both the costs and the benefits are usually hard to measure.

What is the cost of writing automated tests? I’ve asked this question before and recorded the answers (inasmuch as we have them) in a previous blog entry. TDD costs us about 10-30% in short-term productivity. The benefit that they found was a reduction in bugs by 30-90%, and a decrease in code complexity by about 30%.

But what about when the costs are higher than the normal 10 to 30 percent? One good example of this is when there’s no test framework for the situation you’re testing. This might be a new language or framework that you’re working with. More likely it’s a complex API that you have to mock out. So that could increase the cost of automated testing so as to outweigh the benefits — especially on a short project. I could imagine situations where the cost of writing the mocks could eat up more than the project itself.

Another case where we might consider skipping testing is when we’re more concerned about time to market than quality. This is almost always a mistake. Your code will almost always last longer than you expect. (Remember Y2K?) And if the code lasts longer than you expect, that means you’ll have to deal with bugs that whole time. But we have to work with the information we have at the time we make our decisions. And sometimes that might tell us that time to market is more important than anything.

One final case I can imagine is when a true expert is coding something that they’re very familiar with. I could picture someone like Uncle Bob writing code (in a language that he’s familiar with) without tests just as effectively as I could write code with tests.

But these situations should not be common; they’re edge cases. In almost all real-world cases, TDD is the right thing to do. Don’t forget, TDD is also a design discipline — it helps you design a better API. So keep doing TDD. But as with any practice, don’t do it blindly without considering why we’re doing it. Make sure you understand the costs, and whether the benefits outweigh the costs.

TDD Is Alive And Well

I went to RailsConf this year, and the very first talk was a keynote by David Heinemeier Hansson (DHH), the creator of Ruby on Rails. The TL;DR of his talk was “TDD rarely has value”. He followed up with a blog post the next day, titled “TDD is dead. Long live testing.“, and 2 more posts. I think this line of thought is terribly misguided, and causing more harm than good. This article is my response.

First, I would like to address the good points of the talk. He said that programming is pseudoscience, and that people want to tell us that there’s a secret to being a better programmer. But what it really takes is working hard — reading a lot of code, writing a lot of code, and rewriting a lot of code. He’s right. And I also agree with him that you should forget about patterns for a while when learning to code. Beginners try to throw patterns at a problem instead of letting the patterns emerge where they’re supposed to.

I don’t completely agree that programming is a pseudoscience. In some ways it is, but I think it’s more of a craft. It’s a craft, because there’s a lot of science involved, but there’s also an art to doing it well. And like any craft, you’re always working to get better. So to respond to DHH’s stance that “software is more like poetry than physics”, I think it falls
somewhere in between.

With regard to the software engineering practices we use, there really isn’t much science available, mostly because it’s a soft science. That is, it’s really hard to isolate a single variable when comparing code between projects. And nobody has the time or money to write the same code so many times that the differences would be statistically significant.

So we don’t have much science on TDD. But we do have some. Here’s a collection of several: StudiesOfTestDrivenDevelopment. And here’s one that explicitly looks are the difference between test-first and test-last: Does Test-Driven Development Really Improve Software Design Quality? What do these tell us? They tell us that TDD costs us about 10-30% in short-term productivity; reduces bugs by 30-90%, and decreases code complexity by about 30%. As Code Complete tells us (in section 20.5, with studies to back it up), improving quality reduces development costs. So, like most Agile practices, this is a case where spending a bit more time in the short term leads to time savings in the long term.

The more important lesson in the talk was that you have to do what works best for you and your situation. If TDD doesn’t give better results, then either find out how to make it give better results, or stop using it. As we often say in the Agile world, Agile doesn’t mean that you can stop using your brain. While I think TDD is appropriate in most situations, there are cases where it’s not worth the additional up-front cost. If the most important thing for your project is time-to-market, then not testing might be the right decision for you.

To me, TDD provides a bunch of benefits. First and foremost, TDD is a design discipline. It ensures that I think about how my code will be used before I think about how to implement it. This is very powerful in ensuring that the code is well-written from the perspective of other code using it.

Tested code provides confidence to be able to make changes without breaking things. If we write tests after the code, we’re less likely to write them. Tests written after the code also tend to test the implementation instead of the desired functionality. What we really want is tests written as a specification. With tests as a specification, we can come back later and understand why code was written. Without tests, or with poor tests, we can’t understand why the code is there; if we want to rewrite it, we don’t have the confidence that we’re not missing something. Writing tests first also ensures that we only write the code that is needed to implement the required functionality.

I’m not sure why DHH hasn’t “gotten” TDD. I’m not sure if it’s because he’s  a better coder than average, or if he just thinks in a different way than most of us. I think it’s partly because he doesn’t understand TDD, which he admitted might be the case. And I think he’s conflating TDD and unit testing.

DHH is influential in the developer community, especially those newer to Ruby and Rails. People listen to what he has to say. I was happy to see almost every other speaker made fun of DHH’s ideas, and most of the crowd knew better. But there will be a lot of others who will hear DHH, respect his opinions, and not give TDD the try that it deserves. And that’s sad, because it will lead to an overall reduction in code quality in the world.

Here are some other people’s thoughts on the matter:


Testing Rails Validators

It’s challenging to test Rails custom validators.

I recently had to write a validator to require that an entered date is before or after a specified date.

It didn’t seem like writing the validator would be too difficult – I’ve written custom validators before, and date comparisons aren’t all that tricky. But when it came time to write the tests, I ran into several issues. And since I always try to follow TDD / test-first, I was blocked before I even began.

The biggest issue was the ActiveModel::EachValidator#validates_each API. It’s definitely not a well-designed API. You write your validator as a subclass, overriding validates_each. The method takes a model object, the name of the attribute of the model being tested, and the value of that attribute. You can also get the options passed to the custom validator via the options method. To perform a validation, you have to update the model’s errors hash.

The big flaw in the API is that instead of returning a result, you have to update the model. This needlessly couples the model and the validator. And it violates the Single Responsibility Principle — it has to determine validity of the field, and it has to update the errors hash of the model. This is not conducive to testing. Testing this method requires testing that the side-effect has taken place in the collaborator (model), which means it’s not really a unit test any more.

So to make it easier to unit test the validator, I broke the coupling by breaking it into 2 pieces, one for each responsibility. I moved the responsibility for determining validity to a separate method, which I called errors_for. It returns a hash of the errors found. This simplified the validates_each method to simply take the result of errors_for and update the errors hash of the model:

def validate_each(record, attribute_name, attribute_value)
  record.errors[attribute_name].concat(errors_for(attribute_value, options))

This made it much easier to unit test the errors_for method. This method doesn’t even need to know about the model — only about the value of the attribute we’re trying to validate. We simply pass in the attribute’s value and the options.

So we could write the tests without even pulling in ActiveRecord or any models:

describe DateValidator do
  let(:validator) { DateValidator.new(attributes: :attribute_name) }
  let(:errors) { validator.errors_for(attribute_value, validation_options) }

  describe 'when attribute value is NOT a valid date' do
    let(:attribute_value) { 'not a valid date' }
    it { errors.must_include 'is not a valid date' }

  describe 'when attribute value IS a valid date' do
    let(:attribute_value) { Date.parse('2013-12-11') }
    it { errors.must_be :empty? }

And the errors_for method looked something like this:

def errors_for(attribute_value, options)
  unless attribute_value.is_a?(Date)
    return [options.fetch(:message, "is not a valid date")]

Integration testing can also be a bit of a challenge. I recommend following the example from this Stack Overflow answer. Basically, create a minimal model object that contains the field and the validation. Then test that the model behaves like you expect with different values and validations.