threehv

One of the problems of being at a startup is the overwhelming amount of work. So much to do, so little time!

Luckily, this helpful flow-chart helps you decide what to do next.

I used to work for a company that built a complicated desktop application, let’s call it Roloduck. The original version was written in about 1999-2000 and subsequent versions (including a total rewrite) were built over the first six years of the new millenium. Over that time my job title varied between developer, senior developer, lead architect, project leader, team leader and technical director. Amongst other things.

One of the major trends in software development at that time (at least for desktop applications) was to offer options – make it configurable and you can please everyone. And you can then sell “consultancy days” as it takes two weeks post-installation before the thing is usable, as you tick all the right boxes and choose all the right options. It also means you can say “hmm, well, there are eight different ways to do that, each with different pros and cons” (as opposed to saying “we can do exactly what you want” or “no, sorry, we don’t do that”).

So this was a big complicated software project; large enough that no one person knew it all the way through, with a variety of different architectures in-built (COM objects is how you do this stuff; oh no, now it’s COM+, now it’s ADO-disconnected and so on).

But we were only a small company. At its largest there were probably ten people working on the product at any one time. However, if you read the company communications you could never tell – it was all the bland, marketing-droid, use three-hundred-words-when-one-would-do style. Our email signatures had the sixteen paragraph disclaimer – “this is only intended for the intended recipient and if you are not its intended then wipe your memory immediately and burn your computer to prevent accidental ingestion”. This worked so well, some of our customers were amazed when they found out how many people we actually had – they thought we were a multinational with hundreds of developers across thousands of countries.

Over the years, we would consistently come across a single competitor – let’s call them Crump Builder. We poached our sales director from them, so we had a pretty good idea of what they were capable of and how they worked. We were small fry, they had funding and a team several times bigger than ours. They had also been going longer than us, so they had an even bigger set of preferences, options and modules to choose from. Most of the time, the sales process would come down to a choice between Roloduck and Crump Builder, and in the vast majority of cases, the prospect would choose Crump Builder.

At the time, I thought this was because we were behind them. They had the more polished, more finished product that did more and went further. But, since the demise of Roloduck (the company went bust recently) I’ve had to revise that view. You see, one of Roloduck’s old customers has chosen Crump Builder as the replacement for their now defunct Roloduck system. And she said that Crump Builder simply does not do a lot of the stuff that they need, whereas Roloduck does.

This news amazed me. Why did we consistently lose out to Crump Builder in those countless sales pitches?

Having thought about this, the only reason I can come up with is that we were not playing to our strengths. We were chasing their tails all the time. “We know they do X and we don’t yet; but we will soon”. “Yes, their version is pretty slick and ours is a bit clunky”. We were trying to present ourselves as their equals, equivalent in size and structure, in fact bigger than them in size and structure. But, because we were smaller, we were being found out. Not once, in our competitive analysis did we say “they are strong there, but we are strong here, so we should push this instead”. We should have pitched it as “we are the small guys – meaning you will get personalised service”, “our bit isn’t finished yet, so we can tailor it exactly to how you need it”

This is exactly what we have planned for our new venture. There are only two of us, so you will have the focussed attention of at least 50% of the company. We can’t afford lawyers or committees, so all the personality won’t be mangled out of our communications (bad jokes and all). And as we’re only small we can’t afford any waste, in either resources or time. So we will do what you need in the quickest, most efficient way possible. It will not work for everyone, but it will work fantastically for some. Those are our strengths and we will play to them.

Github aren’t building new gems at the moment as they finalise their move to Rackspace. So what do you do if you’ve got a gem that you would like to make available?

Gemcutter’s the new kid on the block – it works as a set of plugins to the gem command that mean you can simply go `gem push my.gem` and it becomes available to the world. Nice as this is, gemcutter’s still pretty young and is currently hosted off one guy’s S3 account.

So we went back to good old-fashioned Rubyforge. Except, believe it or not, I’d never published to Rubyforge before. And it wasn’t as straightforward as it could have been.

I was using the echoe library to build my gem – which gave me a simple `rake manifest`, `rake release` workflow. All good. When you `rake release` it asks if you want to publish to rubyforge; you say “yes” and go to bed happy.

Or not in my case.

Firstly, you need to actually configure rubyforge. Install the rubyforge gem, then use `rubyforge setup` to put in your username and password. Then use `rubyforge config` to make them stick.

That done, I `rake release`d. And got a “no group_id configured for mygem” error. Eh? Googling seemed no help whatsoever – it was mentioned in a few places but people seemed to have resolved it without actually posting their solutions (or I was being blind).

What you need to do is tell Rubyforge where to put your gem. In our case we already had a project on Rubyforge that the gem could be attached to – so `rubyforge create_package project_name gem_name` did the trick. Otherwise you need to get a project set up to house your gem.

But it still didn’t work. This time complaining about a processor id. I had no idea what I was looking for this time, until I discovered this pastie by Dr Nic. The autogenerated config file (normally in ~/.rubyforge/auto-config.yml) is missing some vital information – namely codes for different processor architectures. I pasted the list into your config file and then tried the `rake release` again. And this time it worked!

processor_ids:
  IA64: 6000
  AMD-64: 1500
  Any: 8000
  Sparc: 4000
  PPC: 2000
  Other: 9999
  Alpha: 7000
  i386: 1000
  UltraSparc: 5000
  MIPS: 3000

So for the sake of posterity – and anyone else who gets “no group_id configured for mygem” or “no processor_id configured for mygem” messages, I thought I should let you know how I got there.

We just had some customers report a bug. Not good. We didn’t get an exception email. All the tests passed. We couldn’t see anything untoward in the log files. But it was there. We could reproduce it, both in staging and in production. Not good at all.

But the weirdest thing was we couldn’t figure out the cause. Well I could see why the code was failing (after adding some extra log messages). But ‘git blame’ said those lines of code were unchanged in twelve months. Why hadn’t people complained before? Why hadn’t we noticed it?

After much hunting through log files we found the point when the feature last worked. It coincided with a deployment. That deployment was our Rails 2.3.4 forms vulnerability fix. And the bug was in a form – a missing form parameter that earlier versions of Rails ignored, the newer Rails was choking on.

But why didn’t the tests catch it?

After more hunting I saw that the Cucumber test that exercised the form didn’t have a “When I press the Update button” step. And the subsequent tests were passing, even though the update button hadn’t been pressed.

So I added the step in and made the feature pass. Then deployed it as an emergency fix.

However, what are the lessons to learn here (as there are always some)?

• Firstly, testing cannot catch everything.
• Secondly, the cracks in your tests are where the bugs are.
• Thirdly, we probably need some sort of peer review for tests. I feel that this is more important than for code, because once the tests are right you can refactor the code without worry.
• Fourthly, you really need to log everything. Absolutely everything. Don’t worry about your huge log files – that’s what `logrotate` is for. Get it written down so that one time when you have an obscure bug, you’ll be able to find it easily.