It happens to the best of us

Monday, October 5th, 2009

We just had some customers report a bug. Not good. We didn’t get an exception email. All the tests passed. We couldn’t see anything untoward in the log files. But it was there. We could reproduce it, both in staging and in production. Not good at all.

But the weirdest thing was we couldn’t figure out the cause. Well I could see why the code was failing (after adding some extra log messages). But ‘git blame’ said those lines of code were unchanged in twelve months. Why hadn’t people complained before? Why hadn’t we noticed it?

After much hunting through log files we found the point when the feature last worked. It coincided with a deployment. That deployment was our Rails 2.3.4 forms vulnerability fix. And the bug was in a form – a missing form parameter that earlier versions of Rails ignored, the newer Rails was choking on.

But why didn’t the tests catch it?

After more hunting I saw that the Cucumber test that exercised the form didn’t have a “When I press the Update button” step. And the subsequent tests were passing, even though the update button hadn’t been pressed.

So I added the step in and made the feature pass. Then deployed it as an emergency fix.

However, what are the lessons to learn here (as there are always some)?

  • Firstly, testing cannot catch everything.
  • Secondly, the cracks in your tests are where the bugs are.
  • Thirdly, we probably need some sort of peer review for tests. I feel that this is more important than for code, because once the tests are right you can refactor the code without worry.
  • Fourthly, you really need to log everything. Absolutely everything. Don’t worry about your huge log files – that’s what `logrotate` is for. Get it written down so that one time when you have an obscure bug, you’ll be able to find it easily.

My cron jobs and rake tasks won’t write to the Rails log file

Monday, June 1st, 2009

A project I’ve been working on was recently moved up to Rails 2.2 (Rails 2.3 migration coming soon, but we wanted to take things one step at a time).

All the tests passed. Poking it on the staging server worked well. On to production and all was good.

Time passed.

Then, the app needed to be moved to a new server. Actually to a cluster. Slightly more nerve-wracking, but some playing around with the ghost gem and DNS settings and the deployment completed.

More time passed.

And then disaster struck. Well, not disaster, but it wasn’t good. Not at all.

The app had a rake task that was cronned to run every night. And it failed silently. We found out the day after and I scanned the production log file looking for evidence of what went wrong. Nowt. It didn’t even look like the rake task had run – but it had, as I had the data in front of me.

And the day after, it failed again. Another scan of the production log revealed nothing. Things were getting serious.

I suspected the cluster deployment. Maybe cron didn’t have permission to write to the log file, even though Passenger did. But everything looked good on that front. Maybe there was something else weird about the cluster setup – all the cron jobs were set to run on one of the app-servers; maybe moving it would make a difference?

To test this, I opened the Rails console and started manually recreating what the rake task was doing, on a different app-server. Which is when I noticed that script/console wasn’t writing to the production log either. What?

After a couple of hours of head scratching, I finally found out that the cluster deployment was not at fault. In fact, it seems to be the move to Rails 2.2 that had happened many weeks before – in particular it appears that the logger object no longer auto-flushes itself after writing.

We added a quick Rails.logger.auto_flushing = 1 to an initialiser and both script/console and the rake tasks wrote to the log as expected. Phew!

Women in Technology

Wednesday, April 29th, 2009

A conversation between me and my (14 year old) daughter:

Me: Are any of your friends “into” computers? You know, writing stuff, making websites, coding?

Daughter: I think [girl's name] is. She does layouts and codes for web stuff.

Me: Really? Interesting. But just her then?

Daughter: Yeah.

Me: So why aren’t more of your friends into that sort of thing?

Daughter: I dunno. Just not that bothered I guess.

Me: What about the boys you know? Are any of them into computers?

Daughter: No, not really. Just [girl's name]. Why are you asking?

Me: Well this guy did a presentation at a conference with some saucy images on the slides. And now a load of people are up in arms saying “it’s this sort of thing that makes women not get involved with technology”.

Daughter: Oh no, it’s nothing like that. It’s just not very interesting is it?

Moving to Coda

Wednesday, April 22nd, 2009

Tomorrow I’m actually going to do it.

I’m going to change the tools that I use to code in Ruby for the first time in four years. You see, I must be one of the very very few Ruby-programming Mac-users that doesn’t like Textmate.

This is pretty much heresy to most Rubyists.

I used to think I didn’t like Textmate because of the single window interface. Multiple windows could be positioned exactly where I want them (that window only has a couple of lines in it so I can keep that tiny and see both that and that other file at the same time). Tabs are all the same size for each file and you can’t keep them arranged as you want them.

But then I bought the MacHeist bundle. And got a little web-developer’s editor called Espresso. I really like MacRabbit’s CSSEdit so was looking forward to trying Espresso. And like it I did. I even got over my tabbed interface problems (maybe nicely styled tabs with a list view beats ugly tabs with a drawer). But there was a problem. Espresso is crap for Rails development (despite Elliott Cable’s ruby “sugar”).

So, having been spoilt by an “integrated” editor, but still not liking Textmate, where could I go? Coda, by Panic, was where. And I have to say I really like it. It’s not perfect, it’s taken me about a week to get my head round how things hang together. But it deals with ruby and rails and is extensible.

Which is enough to make me shell out the $99. And change the way I work pretty fundamentally.

The curious case of beauty in Ruby (or Rails vs Merb part 2)

Saturday, December 27th, 2008

I’m sure you’ve all heard the Rails 3 announcement. When I first found out my initial reaction was “fuck me“. But shortly after I was filled with a feeling of dread and general unease. And I didn’t know why ….

Firstly, a bit of history.

I first tried programming on a Commodore Vic 20, and then after that a C64. C64 BASIC was very simple – if you wanted to do anything beyond PRINT statements you needed to POKE values into registers and control the hardware directly. Great for learning how things actually worked. And, to be fair, I was shit at it.

But I do remember reading an article on a system called “Smalltalk” and its “Object Oriented Programming”. Suddenly, programming made sense. It read a bit like English. You sent messages to the thing that knows how to answer your question. It was like talking to people. You ask Dave a football question. You ask George a music question. Cos Dave knows crap all about music and George knows nothing about football.

But, in those days, Smalltalk cost a fortune; there was no way a child like me could get hold of a Smalltalk environment. So instead, I got hold of Turbo Pascal 6 With Objects (thanks Dad). It was not Smalltalk but it read a bit like English and it had objects. I played about with Turbo Pascal, went to university (where I didn’t do computing but did do some C++) and then got a job doing Delphi (Turbo Pascal for the 90s). This object-oriented stuff really worked for me; I put a lot of effort into writing classes that had really simple public interfaces and with code that read like English. And Pascal (the language underlying Delphi) was great for that. Then I discovered Java, which meant I could write Delphi-like code but with having to deal with memory management. I also discovered PHP, Python and Ruby. None of which clicked with me; dynamic typing made me nervous (and PHP and Ruby seemed a bit ugly).

However, I needed an ORM for a Delphi project and I thought I should try to copy an open source project. Whilst searching I discovered Rails and thought “this is the one to copy”. But a day into my “copy ActiveRecord into Delphi” plan I thought “this is just like Smalltalk”. Why make an inferior copy when I can use something that’s not far off the Holy Grail. Writing an application on Rails had the same effect on me as my original discovery of Smalltalk – it read like English, it felt fantastic. So I gave up on Delphi and became a Rails programmer.

What I liked about Rails was its emphasis on happiness. When I wrote Rails code I felt like I was writing beautiful prose. I would go back and refactor it until it read correctly. This was not like pure Ruby, which was often ugly. No; Rails had this idea about beauty in code that really got me excited. It made me happy. It also made decisions for me – put your code here, test it like this, set up your database this way. But Rails had performance problems – so Merb was born. A ground-up rewrite of many of Rails’ ideas but with an emphasis on configurability and performance.

Maybe it’s the Engine Yard connection (I turned Engine Yard down for a job because it didn’t “feel right”) – and now I work for Brightbox, one of their competitors – but for some reason, every time I tried Merb I just couldn’t get into it. It was weird. Structurally and functionally it was the same as Rails – but it was Rails plus performance plus options. And I didn’t like it. I never got past the tutorials. Merb emphasises clear and understandable code and was tested with RSpec (which I love). Rails is hard to understand and uses Test::Unit (which is ugly). But I love Rails and I can’t get into Merb. I just couldn’t figure out why.

Until today.

Mr Hanson did a blog post on his first piece of Rails-Merb integration. And something stood out at me. As he was describing Merb’s provides/display functionality I noticed that I didn’t really “get it”. provides made sense, but how does that relate to display. Mr H addresses that directly:

There were a couple of drawbacks with the provides/display duo, though, that we could deal with at the same time. The first was the lack of symmetry in the method names. The words “provides” and “display” doesn’t reflect their close relationship and if you throw in the fact that they’re actually both related to rendering, it’s gets even more muddy.

And then he describes the Rails 3 version of the same functionality. Instead of provides/display it becomes respond_to/respond_with. In particular display @users becomes respond_with @users.

It’s only a tiny thing. Logically and functionally, they are exactly the same. But DHH’s version has an emphasis on the words that are used. How they couple together (display/provide versus respond_to/respond_with).

And there is the reason that I was uneasy about Rails 3. What if Rails lost this emphasis on the human factors – how the words mesh together – in the search of performance. Merb is written functionally, Rails is written emotionally – Merb is about performance, Rails is about feelings.

But DHH has made me feel much better about Rails 3 – he has shown that he will take Merb constructs and Railsify them, humanise them. Because, although code is executed by computers, it is written, and more importantly, read by people like me.

If you find this useful then please take a look at some of my other writing – or recommend me on Working with Rails. Cheers.

Writing tests for your controllers improves the design of your models

Saturday, December 20th, 2008

I’ve recently been updating some old code – partly written by someone else, partly written by myself. At the time, I thought I had written this code really well; looking back on it now, it looks awful. Fair enough, I’ve learnt a lot – I want to look back on old code and shudder. But also, there is very poor test coverage on this app and the tests that there are are quite unwieldy due to an over-reliance on fixtures.

So I’ve been reworking them all using RSpec, my fork of RSpec-Rails and my Object Factory (which means I can avoid fixtures).

Most of the work involves writing a spec that mimics the current behaviour (by inspecting the code and trying to match all paths through it), then refactoring the code, using the spec to prove that I haven’t broken it.

But some points have some really horrible code (and lots of it) within the controllers. As you probably know, Skinny Controllers is the Rails way – your application logic belongs in your models (as they are your application) – the controller should just find or create the relevant model, ask it to do something and then render the results.

Because of this, I opted to just rewrite the actions in question.

To do this I started by writing a Cucumber feature describing things from the user’s point of view. Actually writing the steps that match the feature was a lot of work; because Cucumber is a full stack test you have to deal with all the dependencies that your individual action has (for example, are you logged in with the correct permissions with all associated objects created and in the right state?).

Then I wrote a controller spec. Controller specs in RSpec should use mock objects; you don’t really want to test the models, you just want to prove that the controller finds or creates the right model, asks it to do something and renders the correct output at the end.

So a typical spec looks something like this (note that this is not RESTful as it was an existing part of the application that I am about to change):


  it "should process an item" do
    @item = mock_model Item, :work_type => :buy_stuff, :quantity => 2

    on_getting :process_item, :id => '1' do
     Item.should_receive(:find).with('1').and_return(@item)
      @stuff = mock_model Stuff
      @item.should_receive(:process).and_return(@stuff)
    end

    assigns[:stuff].should == @stuff
    response.should be_success
    response.should render_template('admin/orders/process_item')
  end

This basically says:

  • We have an item, of type “buy stuff” with a quantity
  • When the process_item action is called, we expect that the controller will try to find the item with the given id
  • Then we should call process on the item and it should give us some stuff
  • The stuff should be stored in an instance variable, called stuff
  • And a page should be successfully rendered using the process_item template

That’s a pretty succinct explanation of what the controller should do – I can’t think of many ways of making that skinnier. It also bears no resemblance to the actual implementation of the action – which currently looks something like this:


def process_work_item
    @item =Item.find(params[:id])
    case @item.product.class.to_s.underscore.to_sym
    when :buy_stuff
      @stuff = Stuff.build_item(@item)
      @stuff.setup_new
      render :action => :process_stuff
    when :update_stuff
      @item.stuff.prepare_for_update
      render :action => :update_stuff
    else
      render :status => 404, :text => "Item product type #{@item.product.class.to_s} unknown."
    end
rescue ActiveRecord::RecordNotFound
  render_not_found
end

Pretty complicated – and as new types of item are added we need to add more and more clauses to that case statement.

First things first – I’ve said that we should call “process” on the item class. So I add a pending spec to the Item specification – this is to remind me that I’ve got some work to implement later on.


  it "should process itself based upon its work type"

Then I rework the controller so that the controller spec passes.


    @item =Item.find(params[:id])
    @stuff = @item.process
    render :template => "admin/orders/process_item"

Pretty simple huh?

What we have done is shifted the logic from the controller into this new “process” method on the Item. We have made it so that the controller knows virtually nothing about the item – all it knows is how to find it and that it has a process method. All the implementation details are now hidden within the Item, out of the way of the outside world.

Through the use of mocking we can ignore actual implementations and concentrate on presenting ourselves as simply, and minimally, as possible to the outside world. This reduces coupling, increases flexibility and makes our code easier to read. Don’t you agree?

Acceptance Testing in Ruby, Rails, RSpec and Cucumber

Friday, November 21st, 2008

I’ve written up a new post at the Brightbox blog detailing how we are using RSpec and Cucumber to build acceptance tests for the next generation Brightbox systems.

Rails 2.1 and 2.2: CSRF vulnerability and work-around

Wednesday, November 19th, 2008

I’ve done a quick write-up on the recent CSRF vulnerability on the Brightbox blog.

Rails vs Merb (updated)

Tuesday, November 18th, 2008
Ruby Programmers having a fight

Ruby Programmers having a fight

What the fuck is this?

Merb is launched and DHH suddenly has a load of “Rails Myths” posts up on his blog. Like this sly little dig:

it shows the great power of being an full-stack framework

Wycats responds with a slightly less sly dig:

For the moment, these differences are the reason that Rails will continue to dominate amongst developers seeking to build apps similar in scope to apps built by 37Signals. I suspect that Merb will pick up steam amongst developers looking to build innovative apps leveraging the latest and greatest Ruby techniques and libraries.

Zed responds angrily to a mistake by DHH (which DHH subsequently corrects).

I admit I’ve not had much time to look at Merb in detail; the times when I have played with it my impression has been “it’s much the same as Rails but done in a different (probably cleaner) way”. I love the fact in Rails that everything comes in one bundle (apart from RSpec :-), I love the fact that Merb gives you choices (even though I don’t have the time to research those choices), I like the fact that the two frameworks are now feeding off each other.

But the thing that impressed me most when I came to Rails was how nice and friendly the Ruby community was. But, it would appear that that was an illusion and massive egos are in charge. Discussion is good. Adapting your ideas in the face of competition and change is good. Having a massive pissing match because my framework is better than your framework is stupid. I wish you would all just shut the fuck up.

UPDATE:

And the fighting continues – this time it’s _why versus Zed.