But the weirdest thing was we couldn’t figure out the cause. Well I could see why the code was failing (after adding some extra log messages). But ‘git blame’ said those lines of code were unchanged in twelve months. Why hadn’t people complained before? Why hadn’t we noticed it?

After much hunting through log files we found the point when the feature last worked. It coincided with a deployment. That deployment was our Rails 2.3.4 forms vulnerability fix. And the bug was in a form – a missing form parameter that earlier versions of Rails ignored, the newer Rails was choking on.

But why didn’t the tests catch it?

After more hunting I saw that the Cucumber test that exercised the form didn’t have a “When I press the Update button” step. And the subsequent tests were passing, even though the update button hadn’t been pressed.

So I added the step in and made the feature pass. Then deployed it as an emergency fix.

However, what are the lessons to learn here (as there are always some)?

• Firstly, testing cannot catch everything.
• Secondly, the cracks in your tests are where the bugs are.
• Thirdly, we probably need some sort of peer review for tests. I feel that this is more important than for code, because once the tests are right you can refactor the code without worry.
• Fourthly, you really need to log everything. Absolutely everything. Don’t worry about your huge log files – that’s what `logrotate` is for. Get it written down so that one time when you have an obscure bug, you’ll be able to find it easily.

From the customer’s point of view it’s pretty essential. You need to know how much you are spending before the work begins so you don’t get stung.

From the developer’s point of view it’s pretty difficult to do because you don’t know how long things will take until you know what you’re building. And you don’t know the details of what you are building until they become apparent, which is normally while you are doing the work.

For this particular client I wanted to get this piece of work out of the way as quickly as possible and the requirements were deceptively complicated. As trying to deal with intangibles is a recipe for disaster, and time was short, I was pretty sure that whatever estimate I came up with would be totally inaccurate.

So to deal with the intangibles I thought I may as well get stuck in with the actual work. And what’s the first thing that I do? Write a cucumber story.

I went through the requirements documents (which after weeks of to and fro were actually quite detailed and refined) and broke it down into four features, each with a number of scenarios. Each scenario was then broken down into individual steps – however, I did not write any step definition (ruby) files. This is cucumber as customer documentation, not as integration testing.

We agreed that the features met the requirements (the client was very impressed with the level of detail and the clarity that cucumber offered) and I simply went through each feature, scenario and step and came up with a price for the step individually. It’s still guesswork, but it’s guesswork on a small scale, so you’re less likely to be way off. Total the lot and there’s your estimate.

And even better, once the client has agreed the price and you’ve started development, you’ve got a development plan, a measure of progress and proof that it all works – all just a simple rake features away.

Image by lemon drop

Spanner in the works

If you are just using plain old Webrat you can pepper your code with puts statements so you can check the value of variables, the existence of HTML elements and the flow of code as it happens. But with Selenium or Watir, you need to run your app separately to Cucumber, normally in a hidden, background, process, so the output of your puts statements is lost in the ether (or an empty pipe).

After having a particularly annoying and hard to trace bug, that was related to an interaction between form content and javascript, I came up with an extremely simple debugging tool.

Just add the following into one of your steps files:

When /^I pause$/ do
  STDIN.gets
end

Then, find the feature that is causing you grief and insert a “when I pause” step at the appropriate time.

When I do this
And I do that
And I pause
And I press "Save"
Then I see my newly created object


Cucumber will power your app, poking it until it gets to the “when I pause” step. It will then pause, waiting on STDIN for you to hit return – giving you time to open your inspector window and poke around in the form as the tests see it.

In this particular case, my steps file had an incorrectly named element within it – all it took was an inspection of the element in question and I saw the error. Hours of frustration wiped out by one of the simplest commands there is.

Spanners by woodsy

I started off by thanking him for his Demeter’s Revenge plugin, which is one of the first things I install on a new project. He said he didn’t use it much any more, as he doesn’t do mocking, except during the design process. This surprised me, but thinking about it, the distinction between what your different types of tests are for is an important one.

The most important part of using mocks is their value in designing your class’s public API. You start with a cucumber feature (your acceptance test) and as you are implementing it, fill in specs for each component in turn. As features are defined in terms of the user interface, you start by specifying and designing your view and controller. The controller needs to interact with a model. And this is where mocking comes into its own.

describe WotsitPokesController
  it "should poke a wotsit" do
    on_posting_to :create, :id => '1' do
      @wotsit = mock_model Wotsit
      Wotsit.should_receive(:find).with('1').and_return(@wotsit)
      @wotsit.should_receive(:poke).and_return(true)
    end
    response.should redirect_to(edit_wotsit_path(@wotsit))
    flash[:notice].should == 'Your wotsit has been poked'
  end

(this uses the RSpec-Rails Extensions to tidy up the specification)

Whatever @wotsit.poke does, when you are mocking, it is in your interests to encapsulate its behaviour within a single method. Otherwise you end up writing tons of “fake” code within your spec, which makes the tests brittle and hard to maintain.

A small housekeeping note; as soon as your controller spec references @wotsit.poke, you need to go to your model spec and add:

  it "should poke itself"

A pending specification, just as a reminder that you’ve got a method to implement in a few minutes time.

But Luke had started doing full-stack testing alone; using shoulda (with its nested contexts) to write acceptance tests, rather than RSpec and mocking.

I’m not so sure about this. His reason was the brittleness of mocked tests. Agreed, it can be dangerous; if you rename the “poke” method on your Wotsit, your controller spec will still pass. But, crucially, your cucumber story (or shoulda integration test) will not.

So I guess the point to make here is that specifying with mocks alone can be a bad idea. You need a full-stack test to catch the stuff that “falls between the cracks”, the proof that your application does what it is supposed to do.

Can you just forego the mocks completely? I don’t think so; they are too valuable during the exploration/design phase.

Should you just get rid of the mocks once your acceptance test is passing? Luke says yes. I’m still undecided, but certainly am tending towards no.

So, what’s your take? Are mocks too brittle to be used in long term development? Or, coupled with an acceptance test, is the value they give so great we need to keep them in our test suite?

(interestingly, since I drafted this post, Pat Maddox has also written about much the same subject with a similar outcome – including Matt Wynne stating “Acceptance tests are for regression coverage, unit tests are for design” in the comments, although Pat himself says you can delete the specs once the design is done).

There are a number of issues with using WATIR – the main one being speed – as WATIR scripts the browser, an extensive test suite will take a long time to run.

But my first hurdle was actually getting the code running. Unlike using Cucumber with Webrat, where your Cucumber steps have access to your Rails code (so you can play around with the response object), with WATIR you run the application in a separate process and then poke the browser in the same way a real user would. Of course, unlike Webrat, you get to test your Javascript too.

However, I didn’t want to have to remember to start my application before running the tests (and probably forgetting to put it into the test environment or running on the wrong port) – so I came up with the following for your features/support/env.rb.

system 'rake db:test:clone'
system 'rm log/test.log'
system 'ruby script/server -p 3001 -e test -d'

at_exit do
  system "kill `ps aux | grep mongrel_rails | grep -e '-p 3001' | grep -v grep | awk '{ print $2 }'`"
end

Before do
  @browser = Watir::Safari.new
  @browser.set_fast_speed
  @base_url = 'http://localhost:3001'
end

This clones your development database, gets rid of the test log (as we’ll need to look through that to track down any errors, so we want to keep it small) and then starts your app in test mode on port 3001 (so it doesn’t clash with the version you’ve already got open in development). The Before clause sets up WATIR and configures a base url that my steps use elsewhere. And the at_exit hook looks for the server we started and kills it.

Next step is to add a platform instruction – so you can choose to test with Safari, Firefox (if I can ever get the Firewatir plugin installed) and Internet Explorer (where I guess the kill statement will need a bit of alteration).

An alternative is to tell mongrel to start and write a pidfile to a known place, and then replace the ps/grep line with:

  pid = File.read(File.expand_path(RAILS_ROOT + "/log/mongrel.pid")).chomp
  `kill #{pid}`

Thanks to Caius for that version.