Welcome to part two of the the post series which will hopefully cauterize the bleeding stump that is my Ruby Testing Quest.
Let’s get going.
A simple RSpec example
Here’s a simple RSpec example.
[snip 'coderspec_example' cannot be found]
This is obviously extremely dull and pointless – just like the minitest one – but it contains just enough to exercise the major parts of RSpec that I care about. It’s actually slightly more sophisticated than the example that I used for MiniTest, because RSpec provides a couple of notable features that MiniTest doesn’t provide. Specifically, these are
before :all setup blocks, and nested groups of tests12.
I’m not particularly interested in looking at the other obvious distinguishing features of RSpec, like matchers and the BDD-style “should” language, as these aren’t actually a part of the core RSpec implementation3.
The two hallmark attributes here that I am interested in are:
- grouping test definitions within blocks (as opposed to classes)
- defining test behaviour using blocks (as opposed to methods)
The simplest way of running this spec would be to save as
something_spec.rb and run it from the command-line.
$ ruby something_spec.rb .. Finished in 0.00198 seconds 2 examples, 0 failures [Finished in 0.5s]
So – what’s actually happening here?
As with the minitest example, the first line loads a special file within the test library that not only loads the library, but also installs an
at_exit hook for Ruby to run when the interpreter exists.
In RSpec’s case, this is defined in
RSpec::Core::Runner.autorun. This calls
ARGV and the
In contrast with MiniTest, RSpec parses the options at this point, and will try to determine whether or not to launch using DRb. In most cases it will create an instance of
RSpec::Core::CommandLine with the parsed
options, and then calls
run on that instance.
run method, some setup happens (mostly preamble to be output by the
reporter, which is set via the configuration). Then we iterate through all of the “example groups”, returned by
Let’s take a diversion to see how things actually get into
Your example groups
Consider our example spec again. At the top we have a call to
describe method is actually defined within the module
RSpec::Code::DSL, but this module is extended into
self at the top level of the running Ruby interpreter (which is
main, a singleton instance of
Object), making the methods in that module available to call in your spec files. You can actually see all of the modules that have been extended into this instance:
[snip 'codemain' cannot be found]
From this we can tell that the ancestors of
Object are still just
BasicObject, but the ancestors of the specific instance
main includes a few extra modules from RSpec. Anyway, moving on…
describe method in
RSpec::Core::DSL passes its arguments straight through to
RSpec::Core::ExampleGroup.describe. This is where things get a little interesting. Within this inner
describe method, a subclass of
RSpec::Code::ExampleGroup is created, and given a generated name.
[snip 'codeexample_group_constants' cannot be found]
The class that was created is there:
Nested_1. For each
describe at the top level, you’ll have a new generated class:
[snip 'codeexample_group_constants_2' cannot be found]
After each subclass is created, it is “set up” via the
set_it_up method, which roughly speaking adds a set of metadata about the group (such as which file and line it was defined upon, and perhaps some information about the class if it was called in the form
describe SomeClass do ...), and stashes that within the created subclass.
More importantly, however, the block which was passed to
describe is evaluated against this new subclass using
The effect of using
module_eval against a class is that the contents of the passed block are evaluated essentially as if they were within the definition of that class itself:
[snip 'codemodule_eval' cannot be found]
You can see above that the behaviour is effectively the same as if we’d defined the
hello? method within the
Lionel class without any “metaprogramming magic”5.
It’s because of
module_eval that you can define methods within example groups:
[snip 'codemethod_in_example_group' cannot be found]
These methods are then effectively defined as part of the
Nested_1 class that we are implicitly creating. This means that methods defined in this way can be called from within your specs:
[snip 'codemethod_in_example_group_called_in_spec' cannot be found]
We’ll see how this actually works a bit later. Knowing that the contents of the
describe block are effectively evaluated within a class definition also explains what’s happening when the
before methods are called:
Because this is evaluated as if it was written in a class definition, then
before must be a method available on the
ExampleGroup class. And indeed it is –
before method actually comes from the module
RSpec::Core::Hooks, which is extended into
ExampleGroup. RSpec has a very complicated behind-the-scenes hook registry, which for the purposes of brevity I’m not going to inspect here..
before method registers its block within that registry, to be retrieved later when the specs actually run.
Because I’m not going to really look too deeply at hooks, the call to the
after method works in pretty much the same way. Here it is though, just because:
The spec itself
The next method that’s
module_eval‘d within our
ExampleGroup subclass is the
Users of RSpec will know that you can call a number of methods to define a single spec:
example, and others with additional meaning like
focus. These methods are actually all generated while RSpec is being loaded, by calls to
define_example_method within the class definition of
ExampleGroup. For simplicity’s sake (pending and focussed specs are somewhat outwith the remit of this exploration), we’ll only look at the simplest case.
it is called, more metadata is assembled about the spec (again, including the line and file), and then both this metadata and the block are passed to
RSpec::Core::Example.new, which stashes them for later.
Within our outer example group, we’ve nested another group:
Just as the top-level call to
describe invokes a class method on
RSpec::Core::ExampleGroup, this call will be invoked against the subclass of
Nested_1) that our outer group defined. Accordingly, each call to
describe defines a new subclass6, stored as a constant within the top-level class:
Nested_1::Nested_1. This subclass is stored within an array of
children in the outer
Within the definition, our
it calls evaluate as before.
Your spec, as objects
So, for every
describe, a new subclass of
ExampleGroup is created, with calls to
after registering hooks within that subclass, and then each
it call defines a new instance of
RSpec::Core::Example, and these are stored in an array called
examples within that subclass.
We can even take a look at these now, for a simplified example:
[snip 'codeexample_group_object' cannot be found]
Where example groups are nested, further subclasses are created, and stored in an array of
children within their respective parent groups.
Phew. The detour we took when looking at this aspect of minitest was much shorter, but now that we understand what happened when our actual spec definition was evaluated, we can return to RSpec running and see how it’s actually exercised.
As we saw above, the
describe method returns the created subclass of
RSpec::Core::ExampleGroup, and when that is returned back in
register method is called on it. This calls
world.register with that class as an argument, where
world is returned by
RSpec.world and is an instance of
RSpec::Core::World, which acts as a kind of global object to contain example groups, configuration and that sort of thing.
register on the
World instance stashes our
Nested_1 class in an
example_groups array within that world.
Our diversion is complete! You deserve a break. Go fetch a cup of your preferred delicious beverage, you have earned it!
Back in RSpec
OK, pop your brain-stack back until we’re in
RSpec::Core::Commandline#run again. Our reporter did its preamble stuff, and we were iterating through
@world.example_groups, whose origin we now understand.
For each example group, the
run method is called on that class, with the reporter instance passed as an argument.
This gets a bit intricate, so I’m going to step through the method definition itself (for version
2.12.2) to help anchor things.
RSpec has a “fail fast” mode, where any single example failure will cause the execution of specs to finish as quickly as possible. Here, RSpec is checking whether anything has triggered this.
reporter is notified that an example group is about to start. The reporter can use this information to print out the name of the group, for example.
The run of the examples is wrapped in a block so it can catch any exceptions and handle them gracefully as you might expect.
before :all hooks
The call to
run_before_all_hooks is very interesting though, and worth exploring. A new instance of the example group is created. It is then passed into this method, where any “before all” blocks are evaluated against that instance, and then the values of any instance variables are stashed.
Consider our original example:
Given this, we’ll stash the value of
@shared_thing (and the fact that it was called
@shared_thing) for later use.
[snip 'codeinstance_variable_set_example' cannot be found]
As you can see above, we can poke around with the innards of objects to our heart’s content. Who needs encapsulation, eh?
Why did RSpec have to create an instance of the example group class, only to throw it away after the
before :all blocks have been evaluated? Because RSpec needs to evaluate the block against an instance of the example group so that it has access to the same scope (e.g. can call the same methods) as any of the specs themselves.
Running the example
Now we’re finally ready to run the examples:
To understand this, we need to look at the definition of
[snip 'coderspec_example_group_run_examples_definition' cannot be found]
This method iterates over each
Example that was stored in the
examples array earlier, filtering them according to any command-line parameters (though we are ignoring that here). The most relevant part for us lies in the middle:
A striking parallel with MiniTest
Another new instance of the
ExampleGroup subclass is created. Remember, RSpec created one instance of the class for the
before :all blocks, but now it’s creating a fresh instance for this specific spec to be evaluated against.
Thinking back to how MiniTest works, there’s a striking parallel: where MiniTest would instantiate a new instance of the
MiniTest::Unit::TestCase for each test method, RSpec is creating a new instance of the
ExampleGroup subclass to evaluate each
Example block against.
Instances of this class are used so that any methods defined as part of the spec definition are implicitly available as methods to be called in the “setup” and “test” bodies (see the
module_eval section above). Not so different after all, eh?
Next, the instance variables that we stashed after evaluating the
before :all blocks are injected (effectively using
instance_variable_set as we saw above) into this new instance, which will allow the spec to interact with any objects those blocks created. It also means that these values are shared between every spec, and so interactions within one spec that changed the state of one of these instance variables will be present when the next spec runs.
#run method on the
Example subclass is called, passing the
ExampleGroup instance and the reporter. Down one level we go, into
The spec finally runs
Here’s the full definition of
[snip 'coderspec_example_run_definition' cannot be found]
For our purposes, we again only need to consider a small part. Once all the reporter and “around” block housekeeping has taken place, the essential core of the example is run:
The call to
run_before_each introspects the hook registry and evaluates every relevant
before hook against the
ExampleGroup instance. In effect, this will find any
before blocks registered in this example group, and then any blocks registered in any parent groups, and evaluate them all in order, so that each nested
before block runs.
Then, the spec block (stored in
@example_block) is evaluated against the
ExampleGroup instance. This is where your assertions, or matchers, are finally – finally! – evaluated.
If there was a problem, such as a matcher failing or an exception being raised, then the exception is stored against this
Example for later reporting. Just as MiniTest assertions raise an exception when they fail, RSpec matchers raise an
RSpec::Expectations::ExpectationNotMetError exception. It seems this is the universal way of halting execution when a test fails7. Another hidden similarity between RSpec and MiniTest!
As in MiniTest, whether or not the spec failed or an exception occured, an
ensure section is used to guarantee that
run_after_hooks is called, and any teardown is performed.
After the specs have run
Once all the specs in this example group have run, all the examples in any subclasses are run (recall that the inner
describe stashed the nested
ExampleGroup subclass in an array called
children). We map each
ExampleGroup subclass to the result of calling
run on it, which starts this whole process again, for every nested example group. Whether or not this group passed or failed overall is then determined using simple boolean logic:
As we leave the call to
ExampleGroup#run, we run any corresponding
after :all blocks, and also clear out our stash of
before :all instance variables, because they are no longer necessary.
You can once again pop your brain-stack back until we’re in
Having run all of the example groups, RSpec will do a little bit of tidy up, and finally return back up through the stack. Along the way printing the results of the run to the console is performed, before interpreter finally, properly quits.
Phew. You deserve another rest.
Testing with blocks
In contrast to the class-based implementation with MiniTest, we’ve now seen how a block-based test framework can work. In a nutshell, it can be characterised in a couple of key ways:
- the stashing of behaviour blocks, later evaluated using
instance_evalagainst clean test-environment instances (see this section of the MiniTest article for what I mean by “test environment”);
module_evaland subclassing to ensure method definition matches programmer expectation.
I would say these two aspects are the hallmark attributes of an RSpec-style test framework. The other notable aspect is the ability to nest example groups, and the subsequent necessity to be able to gather the implicit chain of setup blocks and evaluate them against the test environment instance, but this could be considered another example of using
Supporting method definition in example groups
One thing I’ve found particularly interesting is that RSpec ends up generating classes and subclasses behind the scenes. I believe this is almost entirely a consequence of wanting to support the kind of “natural” method definition within group bodies (see the
module_eval section again).
If any test framework chose to not support this, there’s almost certainly no reason to create classes that map to example groups at all, and the setup and test blocks could be evaluated against a bare instance of
Supporting nesting and dynamic composition
It’s clear that RSpec has more “features” (e.g. nesting,
before :all and so on) than MiniTest (ignoring the many extensions available for MiniTest, the most sophisticated of which end up significantly modifying or replacing the
MiniTest::Unit.run behaviour). I’m deliberately ignoring features like matchers, or a built-in mocking framework, because what I’m most interested in here are the features that affect the structure of the tests.
It’s certainly possible to implement features like nesting using subclasses and explicit calls to
super, but this is the kind of plumbing work that Ruby programmers are not accustomed to accepting. By separating creation of tests from Ruby’s class implementation, the implicit relationships between groups of tests can take this burden instead, and behaviours like
before :all, which have no natural analogue in class-based testing, are possible.
Now, you may believe that nesting is fundamentally undesirable, and it is not my present intention to disabuse you of that opinion. It’s useful (I think) to understand the constraints we accept by our choice of framework, and I’ve certainly found my explorations of MiniTest and RSpec have helped clarify my own opinions about which approach is ultimately more aligned with my own preferences. While I wouldn’t say that I’m ready to jump wholesale into the RSpec ecosystem, I think it’s fair to say that my advocacy of class-style testing frameworks is at an end.
RSpec and Kintama
I started this exploration because I wanted to understand the relationship between the software I have accidentally produced and what’s already available. I already had strong suspicions that any block-based testing implementation would converge on a few common implementation decisions, and while I have now identified a few interesting (to me) ways in which RSpec and Kintama diverge, the essential approach is the same.
In the final article in this triptych (coming soon, I hope), I’ll walk through Kintama and point those out.
There’s no built-in way to ‘nest’ test groups with MiniTest, or test-unit; the closest simulation would be to create subclasses, and explicitly ensure that
superis called within every
There are other RSpec features like shared examples and global before/after hooks that are definitely interesting, but I need to keep the scope of this article down…↩
I’m not sure why some people prefer the syntax
Module.method; as I understand it they are exactly the same, but the former seems more confusing to me, since if you don’t notice the lower-case w in
worldthen you’d assume it was refering to a constant.↩
It’s not really magic, and it’s not really “metaprogramming”, because it’s all just programming. It just so happens that it’s quite sophisticated programming.↩
The nested class is a subclass of the outer subclass of
ExampleGroup(sorry, I realise that’s confusing), precisely such that any methods defined in the outer class are also available in nested subclasses via the regular mechanisms of inheritance.↩
Raising an exception might not be the only way to stop a test executing at the point it fails; it could be possible to use fibers/continuations to “pause” failing tests…↩
comments powered by Disqus