Driving Selenium from the RSpec Story Runner (RBehave)

[Update] The story runner is now included in the 1.1 release of RSpec, so a lot of the hackery mentioned below is no longer required. See http://rspec.info/ for details.

Background

A while ago I cobbled together some code to drive Selenium from Exactor (which was the acceptance testing framework we were using at the time). That project was offshored shortly afterwards, and I’m pretty sure Selenium never actually got integrated into the build (sneaking a peek at the continuous integration server reveals that even the pre-Selenium acceptance test build hasn’t run successfully for months), but I was convinced of the value of Selenium for testing non-trivial web applications.

On my next project (the Web21C SDK portal) we used Selenium on Rails heavily for automated acceptance testing. This worked well, although the fact that the tests are deployed with the application limits what you can do – you can’t interrogate the database after a test, for example, or dynamically set up stubs for systems you interface with (unless you also deploy the stubs as part of the application, and drive them from the browser).

With Mojo there’s an additional complication: as well as being accessible through a browser, most functionality is also available as a RESTful API, using digest-based authentication. To keep our acceptance tests in one place, we wanted to be able to drive a browser using Selenium, alongside other tests which talked to the server directly. The obvious answer was to use Selenium Remote Control, and I also liked the look of RBehave, which has now been incorporated into RSpec (but not released yet).

The RSpec story runner

The story runner in the upcoming release of RSpec is an evolution of Dan North’s RBehave, which in turn is based on JBehave (from the same author). In the past couple of weeks, David Chelimsky has done some excellent work on extracting the text of stories from the code, leading to the plain text story runner.

Read on for the gory details of my solution to running Selenium tests from the story runner.

Putting it all together

Here’s what I did to get Selenium working from the story runner (you need to check out RSpec trunk – I’m currently using r2791 of rspec and r2814 or rspec_on_rails).

I’m sure when the next release of RSpec comes out a lot of this will be unnecessary, and there will probably be standard places for the stories and ways of running them, but for now I’ve just hacked something together that’s simple and works. Apologies for any pieces of string or duct tape that are still showing!

The files described below are laid out as follows, relative to the root of the Rails project:

+-- lib
| +-- rspec/
| +-- rspec_on_rails/
| +-- selenium-ruby-client-driver/
| | +-- selenium.rb
| +-- selenium-server/
| | +-- selenium-server.jar
| +-- tasks/
|   +-- acceptance.rake
+-- stories/
| +-- all.rb
| +-- helper.rb
| +-- steps/
| | +-- login.rb
| | +-- selenium.rb
| +-- stories/
| | +-- login.rb
| | +-- login.txt

Here’s an example story:

stories/stories/login.txt

Story: Login
  The front page should contain a login form

  Scenario: Loading the home page when not logged in
    When the user goes to /
    Then the title should be 'mojo'
    And the page should contain the text 'Welcome to mojo'
    And the page should contain the text 'Sign in using your OpenID'
    And there should be a field named 'openid_url'
    And there should be a submit button named 'login', with the label 'Sign In'
  
  Scenario: Logging in with an unregistered OpenID URL
    When the user goes to /
    And the user types 'some.openid.provider' into the openid_url field
    And the user clicks the login button
    Then the page should contain the text 'Could not find OpenID server'
    And the page should contain the text 'Sign in using your OpenID'
  
  Scenario: Logging in successfully
    Given a test user
    When the user goes to /
    And the user types 'http://dummy.openid/' into the openid_url field
    And the user clicks the login button
    Then the page should contain the text 'Hi test'

Each ‘Given’, ‘When’ and ‘Then’ corresponds to a step (an ‘And’ is just syntactic sugar for a repeat of the same type of step). The steps are defined in separate files, which can be included as required (see below). This story uses two sets of steps:

stories/steps/login.rb

steps_for(:login) do
  Given "a test user" do
    User.delete_all
    User.create!(:username => 'test', :openid_url => 'http://dummy.openid/',
        :fullname => 'Test User', :email => 'test@example.com',
        :mobile => '0123456789')
    # ActiveRecord::Base.connection.commit_db_transaction
  end
end

stories/steps/selenium.rb

steps_for(:selenium) do
  When "the user goes to $path" do |path|
    $selenium.open path
  end
  When "the user types '$text' into the $field field" do |text, field|
    $selenium.type field, text
  end
  When "the user clicks the $button button" do |button|
    $selenium.click button
    $selenium.wait_for_page_to_load 5000
  end
  Then "the title should be '$title'" do |title|
    $selenium.title.should == title
  end
  Then "the page should contain the text '$text'" do |text|
    $selenium.should have_text_present(text)
  end
  Then "there should be a field named '$field'" do |field|
    $selenium.should have_element_present(field)
  end
  Then "there should be a submit button named '$name', with the label '$label'" do |name, label|
    $selenium.should have_element_present("//input[@type='submit'][@name='#{name}'][@value='#{label}']")
  end
end

Note the use of placeholder parameters, which are parsed out of the lines in the story. The Selenium steps use a global instance of Selenium::SeleniumDriver, which is created in the helper (again, see below). I originally created this in a “Given a browser” step, but it seemed simpler to just create it up front, before running the stories.

stories/stories/login.rb

require File.expand_path(File.dirname(__FILE__) + '/../helper')
run_story_with_steps_for :selenium, :login

This is just a wrapper for the textual story, which specifies which steps are required.

stories/all.rb

dir = File.expand_path(File.dirname(__FILE__))
require "#{dir}/helper"

Dir["#{dir}/stories/**/*.rb"].uniq.each { |file| require file }

Another simple wrapper, this time to run all stories.

stories/helper.rb

Warning: This file is where all the ugly code’s hiding!

ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/../config/environment")

# Hack to use trunk rspec without affecting the rest of the project
$:.delete_if {|dir| dir =~ /rspec/}
$: << File.dirname(__FILE__) + '/../lib/rspec/lib'
$: << File.dirname(__FILE__) + '/../lib/rspec_on_rails/lib'

# Not sure why, but this is required when rspec_on_rails not running as plugin
require 'spec/rails/matchers'

require 'spec/rails/story_adapter'
require File.dirname(__FILE__) + '/../lib/selenium-ruby-client-driver/selenium'

Dir[File.dirname(__FILE__) + "/steps/*.rb"].uniq.each { |file| require file }

unless $selenium
  $selenium = Selenium::SeleniumDriver.new 'localhost', 4444,
      '*firefox', 'http://localhost:3000/', 10_000
  $selenium.start
end

# Don't add an ActiveRecordSafetyListener, or it'll roll stuff back
class Spec::Story::Runner::ScenarioRunner
  def initialize
    @listeners = []
  end
end

# Runs the story in the file with the same name as the calling ruby file,
# but with a .txt extension instead of .rb, using the specified steps.
def run_story_with_steps_for *steps
  with_steps_for *steps do
    # Pull the filename of the caller out of the stack. Must be a better way.
    run caller[3].sub(/\.rb:.*/, '.txt')
  end
end

# Alias some more ruby-like methods on SeleniumDriver, to make it play
# better with rspec matchers.
module Selenium
  class SeleniumDriver
    alias_method :original_method_missing, :method_missing
    def method_missing method_name, *args
      if method_name.to_s =~ /^has_.*\?$/
        real_method = method_name.to_s.sub /has_(.*)\?$/, 'is_\1'
        if respond_to? real_method
          return send(real_method, args)
        end
      elsif respond_to?(real_method = "get_" + method_name.to_s)
        return send(real_method)
      end
      return original_method_missing(method_name, args)
    end
  end
end

Most of this code doesn't really belong 'loose' in a helper file, but I didn't want to expend too much effort on getting everything working tidily with a pre-release version of RSpec.Hopefully there are enough comments to give a vague idea what's going on, but here are some more notes:

• Lines 5–10: There still seem to be some bugs in trunk RSpec which cause existing specs to fail (or possibly bugs in our specs that don't show up in 1.0.8). I've installed the trunk versions of the plugins under lib, and here I pull the real plugins out of the load path and insert the trunk versions instead.
• Lines 17–21: Create a selenium driver, and open a browser session.
• Lines 23–28: By default, RSpec adds an ActiveRecordSafetyListener to the story runner. This rolls back database changes between scenarios, which is great if your calling your code directly, but obviously means that if you write to the database, the server that Selenium's talking to can't see them. There's probably a cleaner way of disabling it.
• Lines 30–37: A convenience method to remove some code from the story wrappers. Basically just replaces '.rb' with '.txt' and runs that file with the specified steps. However, I can't find a clean way of finding the file name of the code that called a method, so I'm pulling it out of the call stack. Ugh.
• Lines 39–56: The method names in the ruby interface to Selenium RC are rather java-like (eg get_title and is_text_present). I've used a bit of method_missing hackery to wrap these methods with more ruby-like names (title and has_text_present? for those specific cases), which allows us to use rspec matchers like $selenium.title.should == foo and $selenium.should have_title bar.

lib/tasks/acceptance.rake

Finally, I've created some rake tasks to wrap everything up, and also to start and stop the selenium server.

desc "Run the acceptance tests, starting/stopping the selenium server."
task :acceptance => ['selenium:start'] do
  begin
    Rake::Task['acceptance:run'].invoke
  ensure
    Rake::Task['selenium:stop'].invoke
  end
end

namespace :acceptance do
  desc "Run the acceptance tests, assuming the selenium server is running."
  task :run do
    system 'ruby stories/all.rb'
  end

  namespace :selenium do
    desc "Start the selenium server"
    task :start do
      pid = fork do
        exec 'java -jar lib/selenium-server/selenium-server.jar'
        exit! 127 
      end
      File.open SELENIUM_SERVER_PID_FILE, 'w' do |f|
        f.puts pid
      end
      # wait a few seconds to make sure it's finished starting
      sleep 3
    end

    desc "Stop the selenium server"
    task :stop do
      if File.exist? SELENIUM_SERVER_PID_FILE
        pid = File.read(SELENIUM_SERVER_PID_FILE).to_i
        Process.kill 'TERM', pid
        FileUtils.rm SELENIUM_SERVER_PID_FILE
      else
        puts "#{SELENIUM_SERVER_PID_FILE} not found"
      end
    end
  end
end

private

SELENIUM_SERVER_PID_FILE = 'tmp/pids/selenium_server.pid'

Still to do

There's plenty of scope for improvement here. For a start, I haven't figured out how to generate a report for the test run, apart from the plain text output from the console (which is only usable if you start the selenium server in a separate shell). Also, I can't see an obvious way of returning a success/fail status at the end of the run, which makes it difficult to use in a continuous integration process.

The story runner looks like a very promising tool though, and I can't wait for the stable release.

[tags]bdd, rspec, rbehave, selenium, acceptance testing[/tags]