Testing, and the lost art of the manual

Tuesday, January 17, 2012 :: Tagged under: pablolife culture engineering rants essay. ⏰ 8 minutes.


Hey! Thanks for reading! Just a reminder that I wrote this some years ago, and may have much more complicated feelings about this topic than I did when I wrote it. Happy to elaborate, feel free to reach out to me! 😄


Note: I edited a bunch of this language to be a bit more, uh, fair. Looking back, I was very angry! This is also an example of "ignoring Hot New Things is its own skill."

I also regret bringing up Uncle Bob because when he's not denying trans people's struggles or wanting to fuck his computer, he's giving terrible programming advice that amounts to "toughen up" and "but I, personally, don't already do that." The useful bit about the talk he gave was citing Ward Cunningham anyways.


I hate to be GrandPaul, the old curmudgeon who hates things that just ain't the way they used to be. But I've decided to touch webdev for ScrabbleCheat and by jove, I feel those RoR kids have jumped the shark. Forget TDD; that shit is so passe. These kids practice BDD, with Cucumber and RSpec.

Let's make a few thing clear here: Unit Testing isn't the end-all, be-all of automated software verification. BDD seems to be a perfectly noble attempt to address many of its shortcomings, or if I may go so far, addressing the "failed promises" of TDD. Some of those promises (and how BDD helps solve them include):

I bring all this up because, well, I'm trying to get on this train and I'm finding it a bit of a pain in the ass. There are two major reasons for it, one the current Ruby ecosystem in particular, and the other simply being skeptical of the whole approach in the first place.

BDD in Ruby, today

Despite my seething cynicism, I'm working through it and giving it a try. Hell, the only reason I joined Twitter was because, four years ago, I was getting angry at its very existence, feeling that "Web 2.0" had brought us to such a shallow, useless service. Then I realized I was Getting Mad About The Kids, and that path looked doomed to bitterness, so I signed up. 2500 tweets later, I love it. Maybe (and I still believe this) I'll come around and preach this the way the Ruby kids do these days.

But where to start learning? I looked at RoR when it just hit the scene and I'd taken exactly one programmng class, back when we were all still using Test::Unit, I understand MVC, but wanted to do things right this time. So I read about RSpec and Cucumber, go to their respective websites, and try to write my first cukes and specs. The first thing you notice is that they both use DSLs. I know me plenty of Ruby, but when all your examples look like

Feature: An online spittoon which allows users to spit at the monitor for fun and profit

Scenario: Spittoon graphics
Given that my website is operational
And that the user is full of saliva
Then that user may spit upon the screen
But not upon the keyboard
And a bucket will appear to receive the spit

Scenario: Proper spittoon sounds
Given that the user has spit upon the screen
When the user spits on it
Then it should make a great "ding!" sound

It's not obvious where the code comes in. Or what the rules are for writing these files. Similarly, RSpec shows you tons of files like

describe Spittoon do
    subject { :spittoon }
    it "should take spit in the monitor" do
       visit("./")
       subject.should receive_spit
    end
end

All of this is well and cute, so you ask yourself, "where is the DSL reference so I can write my own?"

No really, where is it? Because I still haven't found it. I've looked all through the interwebz, but couldn't find a definitive reference for the DSLs of either the Cucumber libraries I'm using nor RSpec. There are a few half-baked tutorials, some Wikis on the Github pages, and while they all describe these technologies on a very shallow level, they barely answer my questions. When you run rspec from the command-line, where is it looking? Same with cucumber? How should all the components fit into the Rails app? Where is a full specification of the DSL? What capabilities of the browser does Capybara emulate? And where's its DSL reference?

To contrast, there's no shortage of press pages and splash pages telling me how this will add business value, how easy they are to use, how they all play well together, how they'll turn you into a rockstar ninja coder that makes makes clients happy. I'd believe them, but if I want to read how you'll redirect me to a dozen scattered wiki links and shallow blog posts with toy examples.

Well, that's not quite true. Because for $25 you can buy The RSpec Book. Or the Cucumber book, for another $25. Note that many a Rails book carries a list price of $40-$50. Yes friends, you can have a decent serial, instructive reading experience that doesn't involve navigating disjointed wiki tabs if you pay up. All the refrains about how Java was as good for the publishing industry as it was for the software industry come back. PragProg is a house built on their Ruby + Rails books. Never mind that the reviews for the books state that they are mostly out of date (all the examples use Webrat, not Capybara) or also only contain only trivial, toy examples. You can't really blame them, it's part of the limitation of the dead tree business. But you know what can solve this? A good, freely available manual.

I can learn a mainstream language from 1995 (Ruby), I can leverage existing programming experience from several other languages and projects, but when Rails apps use a component list reminiscent of The Startup Guys ("So I'm using Cucumber and RSpec with Rails with a dash of Factory Girl to practice BDD, with Launchy and Capybara since I think TDD and Webrat are dead.") there a real need to ensure that, if you rely on DSLs, the barriers to entry are low and the ability to learn is high.

To compare to another learning experience I'm having, I've picked up emacs. I've been a Vim user for the past 3 years, but felt like I wanted to taste the forbidden fruit and so have started doing all my text editing in Emacs (this post was written in Emacs). I was off the ground, blown away by some of the capabilities in less than a day. I was using SLIME to run excercises in PAIP in less than a day, filled a few TODO lists in org-mode, and found a redundant import using Erlang-mode. And d'you wanna know how I did it?

I downloaded SLIME... and the manual. I downloaded org-mode... and the manual. I downloaded Emacs... and the manual. When I need to know something, it's in there. But now, I'm feeling less friction learning one of the most bloated and feature-packed pieces of software in history than I am trying to get a basic site up in Rails, using the community's idioms and best practices.

At RailsConf 2009, Robert Martin gave a great talk titled "What killed Smalltalk could kill Ruby, too", and he highlights properties of the Smalltalk culture before it died: a bunch of very talented, smug programmers who were sure they would inherit the world because their tools were so much better than the competition. You can ask Lispers how that worked for them, too. But at the end of the day, Rubyists: try not to make too much of a mess, since for every person like me who comes along to join the party, 10 won't stick around past this frustration as I have. Do the dirty work and write a damn manual, preferably downloadable as a PDF.

On BDD, generally

All that being said, I can't help but be skeptical about the advantages of BDD in the first place. A lot of it is well said in this blog post, but I feel like BDD doesn't offer a whole lot of advantages over TDD, and the criticisms of TDD seek a more powerful answer.

I listed two criticisms at the top of the post. Tackling them in reverse order:

Bryan O'Sullivan (of Real World Haskell fame) had a zinger of a comment in an article describing BDD:

Oh wow. I never knew that Rubyists had a Cobol fetish. I think I liked my brain better when it didn't contain that knowledge.

It just feels like so much typing for not much power. "Natural"-sounding executable text doesn't strike me as a critical gain on the weaknesses of TDD.

Despite all my grousing, I'll stick through it. Watch me write a blog post in a few months swearing by it, like my Twitter conversion. Just wanted to get this off my chest.

Thanks for the read! Disagreed? Violent agreement!? Feel free to join my mailing list, drop me a line at , or leave a comment below! I'd love to hear from you 😄