It's been a while, but we've gotten several new things into Machine.Specifications (MSpec). I'm excited to finally release them for everyone to start playing with. You can grab the bits here.

Let's talk about what's new though. Here's an example of a new context/spec:

  [Concern("Console runner")]
  public class when_specifying_a_missing_assembly_on_the_command_line
  {
    Establish context = ()=>
    {
      console = new FakeConsole();
      program = new Program(console);
    };

    Because of = ()=>
      exitCode = program.Run(new string[] {missingAssemblyName});

    It should_output_an_error_message_with_the_name_of_the_missing_assembly = ()=>
      console.Lines.ShouldContain(string.Format(Resources.MissingAssemblyError, 
      missingAssemblyName));

    It should_return_the_Error_exit_code = ()=>
      exitCode.ShouldEqual(ExitCode.Error);

    const string missingAssemblyName = "Some.Missing.Assembly.dll";
    public static ExitCode exitCode;
    public static Program program;
    public static FakeConsole console;
  }

There have been a few semantic changes

• The Description attribute has been removed. There is now an optional Concern attribute that allows you to specify a type and/or a string that the context/spec is concerned with.
• Context before_each is now Establish context.
• Context before_all is now Establish context_once.
• Context after_each is now Cleanup after_each.
• Context after_all is now Cleanup after_each.
• When {...} is now Because of. This is closer to SpecUnit.NET's verbage, and doesn't force you to specify the "when" twice.

There is now a console runner

We don't quite have all the options we want yet, but the basics of the runner are working. Here's the help from the runner:

We also stole Bellware's SpecUnit.NET reporting stuff and ported it over. You can now generate a report on your specs with the --html switch. Here's an example run:

This is the report it generates.

Want to try it out?

1. Grab the drop here.
2. Extract it somewhere. Put it somewhere semi-permanent because the TestDriven.NET runner will need a static location for the MSpec TDNet Runner.
3. If you want TestDriven.NET support, run InstallTDNetRunner.bat
4. Check out the example in Machine.Specifications.Example. Note that you can run with TD.NET.
5. Create a project of your own. Just add Machine.Specifications.dll and get started.
6. Send me feedback! Leave comments, email me, tweet me, whatever.

Also, this is part of Machine, so feel free to take a look at the code and/or submit patches. There's also a Gallio adapter in there, but I didn't include it in the release as it's not quite polished enough yet. If you're interested in it, talk to me. Special thanks to Scott Bellware, Jeff Brown and Jamie Cansdale for their help and support. Also, extra special thanks to Eleutian's newest dev, Jeff Olson for much of the recent work that has gone into MSpec!

Just wanted to quickly note that I tracked down the performance issue in Rhino.Mocks and patched it. I also updated the original post with the new numbers. Enjoy!

I've complained before that Mocking is Slow but I never really dove further into it. Today I decided to actually compare Rhino.Mocks to other mock frameworks on a pure performance basis to see if it was a global problem. I timed 2000 unit tests across 100 classes with 20 tests each. The results were a bit surprising:

That's right, according to these tests, Rhino.Mocks is at least 3 times slower than the other frameworks when under heavy load in TD.NET and five times slower in the console according to these tests. It's also interesting to note that TypeMock is faster than Moq in TD.NET, but slower in the console runner.

While running the Rhino.Mocks tests it is very clear that there is a degrading performance issue. All the other frameworks executed tests with a near constant speed per test, but Rhino.Mocks slowed down noticeably about half way through.

Please feel free to try it yourself, grab the project here. You should be able to just run the 4 strategy .bat files (run-rhino, run-moq, run-tmock-reflective, run-tmock-natural). Let me know if you find anything interesting.

As some of you who follow me on twitter know, I've been working on Yet Another Context/Specification Framework as an experiment. Yeah, I know we already have NSpec and NBehave, and they're great and all, but MSpec takes things on from a slightly different angle, and it's just an experiment (for now). Here's a sample Description:

[Description]
public class Transferring_between_from_account_and_to_account
{
  static Account fromAccount;
  static Account toAccount;

  Context before_each =()=>
  {
    fromAccount = new Account {Balance = 1m};
    toAccount = new Account {Balance = 1m};
  };
  
  When the_transfer_is_made =()=>
  {
    fromAccount.Transfer(1m, toAccount);
  };
   
  It should_debit_the_from_account_by_the_amount_transferred =()=>
  {
    fromAccount.Balance.ShouldEqual(0m);
  };

  It should_credit_the_to_account_by_the_amount_transferred =()=>
  {
    toAccount.Balance.ShouldEqual(2m);
  };
}

And a TestDriven.NET run:

------ Test started: Assembly: Machine.Specifications.Example.dll ------

Transferring between from account and to account
  When the transfer is made
    * It should debit the from account by the amount transferred
    * It should credit the to account by the amount transferred


2 passed, 0 failed, 0 skipped, took 0.79 seconds.

Err, What?

Different eh? The idea was heavily inspired by Scott Bellware's SpecUnit.Net framework he showed at the ALT.NET conference. It also took heavy cues from RSpec and my insanity. I realize that the the code doesn't look much like C# code and I'm OK with that. Many have and will ask why I don't just use Boo or RSpec w/ IronRuby eventually or even one of the existing Context/Spec/BDD frameworks. Those are good questions, but my main motivations are tooling and syntax. I enjoy the tooling I get in C# and I personally like the syntax in this library considering the limitations imposed by C#.

How's it work?

The simplest way to describe it is to compare it to a normal *Unit style testing framework:

• Description = TestContext
• Context before_each = SetUp
• Context before_all = SetUpFixture
• Context after_each = TearDown
• Context after_all = TearDownFixture
• When = Also SetUp, but happens after Context before_each
• It = Test

Rather than methods and attributes, MSpec uses named delegates and anonymous functions. The only reason for this is readability. You'll also notice that the fields used in the context are static. This is necessary so that the anonymous functions in the field initializers can access them. Probably the first thing you noticed is the =()=> construct. I won't mention the names that this was given on twitter, but I think it's an acceptable thing to have to deal with in exchange for the cleanliness of the rest of the syntax.

Ok, you're crazy, but how do I try it?

First, this is a very rough cut. Everything is subject to change as we experiment with the language. That said, here's how you play with it:

1. Grab the drop here.
2. Extract it somewhere. Put it somewhere semi-permanent because the TestDriven.NET runner will need a static location for the MSpec TDNet Runner.
3. If you want TestDriven.NET support, run InstallTDNetRunner.bat
4. Check out the example in Machine.Specifications.Example. Note that you can run with TD.NET.
5. Create a project of your own. Just add Machine.Specifications.dll and get started.
6. Send me feedback! Leave comments, email me, tweet me, whatever.

Also, this is part of Machine, so feel free to take a look at the code and/or submit patches. There's also a Gallio adapter in there, but I didn't include it in the release as it's not quite polished enough yet. If you're interested in it, talk to me. Special thanks to Scott Bellware, Jeff Brown and Jamie Cansdale for their help and support.

Currently in Rhino.Mocks, making mocks fire events and ensuring that an event on your SUT was fired are both awkward and verbose at best. Here is an example of both things at once:

[Test]
public void ViewFiresBeginDrag_Always_FiresChangedEvent()
{
  IEventRaiser raiser;
  bool eventFired = false;
  using (Record)
  {
    The<ICardView>().BeginDrag += null;
    raiser = LastCall.IgnoreArguments().GetEventRaiser();

    Target.Changed += (x,y) => eventFired = true;
  }

  using (Playback)
  {
    raiser.Raise(The<ICardView>(), EventArgs.Empty);
    Assert.IsTrue(eventFired);
  }
}

Nice eh? First, is there a better way to do either of these things that I'm missing? Please tell me if so. Next, if not, what can we do to clean this up?

Well, Ayende and I discussed this in the past and Ayende spiked it and asked for feedback. The feedback was mixed and for one reason or another it never made it into Rhino.Mocks that I know of.

Well, today while I was working on Machine.Testing and another side project, I decided to give something else a shot. What I ended up with is this:

[Test]
public void ViewFiresBeginDrag_Always_FiresChangedEvent()
{
  using (Record)
  {
    PrimeEventFiringOn<ICardView>(x => x.BeginDrag += null);
    Target.Changed += NewEventFireExpectation<EventHandler>();
  }

  using (Playback)
  {
    FireEventOn<ICardView>(EventArgs.Empty);
  }
}

Better, but I still don't think it's perfect. Also, it probably requires some explanation, so let's pick it apart piece by piece:

    PrimeEventFiringOn<ICardView>(x => x.BeginDrag += null);

=> +=??? Ugly huh? I really wish we could just refer to an event somehow without having to do +=/-=. At least I'm not using vb though... Alas, we cannot access them easily so we're stuck hacking away like this. So this particular method will basically get the mock or stub ICardView.BeginDrag ready to be fired. This needs to be done during the record phase it seems. You can only prime one event at a time per mock, so if you need to do more than one you can revert to the normal Rhino.Mocks syntax.

    Target.Changed += NewEventFireExpectation<EventHandler>();

This was a fun method. This method actually creates a new DynamicMethod in the signature required by the event and creates a new delegate. The method tracks whether or not it was fired, and at the end of the Playback phase in my fixture it will assert that all of the events were actually fired.

    FireEventOn<ICardView>(EventArgs.Empty);

This actually fires the event we set up in the Prime call. It can only be called after you've primed an event.

So, the whole thing is kind of "magic", but it's less code if you can accept the magic. I think we can make things even better though, but it'd require changes to Rhino.Mocks and it's time for bed so maybe Ayende can swing by the Eleutian office while in Seattle and we can work on it. Here's what I'm thinking:

[Test]
public void ViewFiresBeginDrag_Always_FiresChangedEvent()
{
  using (Record)
  {
    Target.Changed += Mocks.CreateEventHandler<EventHandler>();
  }

  using (Playback)
  {
    EventRaiser.Raise(() => The
().BeginDrag += null, The<ICardView>(), EventArgs.Empty); } }

There's probably more you can do with the CreateEventHandler syntax like add more specific expectations, assert its not fired, assert that it's fired X times, etc. The EventRaiser syntax is ugly, but it doesn't involve strings or the fire from the right hand side like the syntax I mentioned at the beginning of the post.

You can get the source here, but I warn you it's first draft and there are hacky bits.

Oh, and here's some example tests using the TestsFor fixture:

[TestFixture]
public class CardPresenterTests : TestsFor<CardPresenter>
{
  private Card _card;

  public override void SetupContainer()
  {
    Override<ICardView>(With.Stub);
  }

  public override void BeforeEachTest()
  {
    _card = new Card(0);
  }

  [Test]
  public void OnBeginDrag_Always_SetsIsInFluxToTrue()
  {
    using (Record)
    {
      PrimeEventFiringOn<ICardView>(x => x.BeginDrag += null);
    }

    using (Playback)
    {
      FireEventOn
(EventArgs.Empty); Assert.That(The<ICardView>().IsInFlux, Is.True); } } [Test]
public void OnBeginDrag_Always_FiresChangedEvent() { using (Record) { PrimeEventFiringOn<ICardView>(x
=> x.BeginDrag += null); Target.Changed += NewEventFireExpectation<EventHandler>();
} using (Playback) { FireEventOn<ICardView>(EventArgs.Empty); } } }

Daniel Cazzulino, author of Moq posted a good comment on my last post where I suggested looking into a Mockito like syntax for .NET Mock Frameworks.

On the surface, Mockito's approach seems good. But if you do the "true" comparison, you'll see that stub(...) is exactly the same as mock.Expect(...) in Moq.

Then, when you do verify(...), you have to basically repeat the exact same expression you put in stub(...). This might work if you only have a couple calls to verify, but for anything else, it will be a lot of repeated code, I'm afraid.

I thought this too. See my comment here from a month ago. Szczepan made a good point and I've thought about it more since then.

When combined with my position on loose vs strict mocks (almost always use loose), I'd say that *most* of the time you are either stubbing or verifying. Meaning, if you're verifying you don't need to stub unless of course that method returns something that is critical to the flow of your test, in which case you don't really need to verify, because the flow would have verified. That's a mouthful, but does that make sense?

I haven't used mockito, and I know there are times I use Expect.Call with return values that matter (which would essentially require you to duplicate stub & verify), but maybe that's a smell? Maybe if you think you need that you can do state based testing or change your API?

Here's an example Test using Rhino.Mocks:

[Test]
public void SomeMethod_Always_CallsSendMail()
{
  IMailSender sender = mocks.DynamicMock
(); UnderTest underTest = new UnderTest(sender); using (mocks.Record()) { Expect.Call(sender.SendMail()).Returns(true);
} underTest.SomeMethod(); mocks.Verify(sender); }

And some code this is testing (obviously not test driven, but you get the idea):

public void SomeMethod()
{
  if (!_sender.SendMail())
  {
    throw new Exception("OH NOS");
  }
}

Notice that here we would need to stub and verify separately with Mockito like syntax. This would look something like this:

[Test]
public void SomeMethod_Always_CallsSendMail()
{
  IMailSender sender = mocks.DynamicMock
(); UnderTest underTest = new UnderTest(sender); Stub.That(() => sender.SendMail()).Returns(true);
underTest.SomeMethod(); Verify.That(() => sender.SendMail()).WasCalled(); }

This may violate DRY, but what if you designed your API differently? Maybe SendMail should throw an exception on failure instead of returning a boolean? This would make the return value unnecessary and remove the need for the Stub call. Clearly you can't always do this, especially with unwrapped legacy or API code, but it's something to think about.

Also, I think you shouldn't be verifying more than one method generally to go along with the one assert/test rule, so a single repeat would not be that horrendous. Heck, you could even do:

[Test]
public void SomeMethod_Always_CallsSendMail()
{
  IMailSender sender = mocks.DynamicMock
(); UnderTest underTest = new UnderTest(sender); Stub.That(var sendMail = () =>
sender.SendMail()).Returns(true); underTest.SomeMethod(); Verify.That(sendMail).WasCalled();
}

I think the syntax would lead to better, more concise tests. But maybe it would just be too annoying? I wouldn't know until I tried it for a while I guess.

(Note: I'm going to speak about .NET mock projects here for the most part, but most of them have Java quasi-equivalents.)

The original mocking frameworks like NMock required you to setup expectations by passing strings for method names. This was fragile and made refactoring more difficult.

A few mock frameworks now allow you to define expectations and mock results in a strongly typed manner. Rhino Mocks and TypeMock use a record/replay method to setup expectations. The record replay method is mostly necessary because the same calls are made on the same objects under two different scenarios. This leads to a few issues.

The first issue is confusion and barrier to entry. Many people have complained that the Record/Replay method is not straight forward and the whole paradigm is confusing. There are also complains about the naming, are you really recording and then replaying? It's just kind of a strange thing. Of course most of us learn to live with it, understand it, and accept it for what it is. Recently though, a few mock frameworks have popped up that do away with this model.

In the .NET world we have Moq. Moq gets rid of the need for record/replay because recordings have a very different syntax. They use lambdas instead of actual calls to the mock object. This allows the framework to know when you are recording an expectation and when you are fulfilling an expectation. It adds a bit of noise in the form of "() =>" but all in all it's not bad. Of course this requires C# 3.0, but it's good to keep looking ahead.

In the Java world we have Mockito. Mockito also does away with the record/replay model but it does it in a different way. At first I wasn't a fan, but thinking about it more, I like it. Mockito has two main apis, stub and verify. Stub is equivalent to SetupResult.For, and verify is equivalent to Expect.Call with a verify. The interesting bit is that the stubbing happens before the the class under test is invoked and the verifying (which includes describing the method to be verified) happens after the class under test is invoked. This is best shown with an example stolen from the Mockito site:

  
  //stubbing using built-in anyInt() argument matcher
  stub(mockedList.get(anyInt())).toReturn("element");
  
  //stubbing using hamcrest (let's say isValid() returns your own hamcrest matcher):
  stub(mockedList.contains(argThat(isValid()))).toReturn("element");
  
  //following prints "element"
  System.out.println(mockedList.get(999));
  
  //you can also verify using argument matcher
  verify(mockedList).get(anyInt());

Obviously it would take a bit of imagination to arrive at a .NET equivalent, but you get the idea. I like this because the normal test structure is Setup Stuff->Do Stuff to Test->Verify Stuff did what it should have. The normal record/replay model requires you to set up verifications before you actually Do Stuff (though you call VerifyAll afterwards). This is a bit less natural. I feel syntax like this (yeah, I like the new NUnit syntax) would be more intention revealing:

Assert.That(() => someMock.Foo(), Was.Called);

Or:

Verify.That(() => someMock.Foo()).WasCalled();

Then you would stub like this:

Stub.That(() => someMock.Bar()).Returns(3);

Note: No idea if this is feasible or makes sense or not, my lambda experience is limited to light reading, but you get the idea. I'm sure the syntax could also be prettier.

Rhino.Mocks is my current mock framework of choice. I'm used to it, I've lightly contributed to it, and I've been working with it for a while now. Despite that, I do think that there is definitely more to explore in the mocking arena especially with C# 3.0.

There are lots of other fun things to talk about too... like TypeMock's magic, but that's another day still...

Moq has been getting some press lately because it's the newest mock framework on the block. I think it's certainly interesting and I'll have more to say on it later, but I wanted to briefly complain about one aspect real quick.

Moq touts that it has a more simplified API when compared to something like Rhino.Mocks. The API has a single entry point which certainly aids discoverability, but I question one of the design decisions. I remember seeing someone say that Rhino had too many types of mocks and that that was confusing. Well, I don't think it has this many different Mock Behaviors:

  public enum MockBehavior
  {
    Strict, 
    Normal,
    Relaxed,
    Loose,
    Default = Normal,
  }

Why have this many? Does anyone know what they do just by looking at them? At least they're documented, but the docs are quite a mouthful:

  public enum MockBehavior
  {
    /// 

/// Causes the mock to always throw /// an exception for invocations that don't have
a /// corresponding expectation. /// 

Strict, /// 

/// Matches the behavior of classes and interfaces /// in equivalent manual mocks:
abstract methods /// need to have an expectation (override), as well /// as all interface
members. Other members (virtual /// and non-virtual) can be called freely and will
end up /// invoking the implementation on the target type if available. /// 

Normal, /// 

/// Will only throw exceptions for abstract methods and /// interface members which
need to return a value and /// don't have a corresponding expectation. /// 

Relaxed, /// 

/// Will never throw exceptions, returning default /// values when necessary (null
for reference types /// or zero for value types). /// 

Loose, /// 

/// Default mock behavior, which equals 

. /// 

Default = Normal, } 

I'm of the opinion that you should only have one type of Mock, and that's what Rhino calls Dynamic and Moq calls Loose. I described why here. If I wanted to simplify mocking, I'd start here.