SpeakENG

Don't ever put a Photo directly in your User entity (or any entity commonly used for anything but displaying a photo). It's not only arguably bad from a normalization standpoint, but NHibernate's flushes will absolutely kill you. Don't say I didn't warn you.

James Kovacs replied to one of my many NHibernate Optimization Ramblings:

In the first example, how do you know that most customers are unimportant until you fetch them from the database? You have an additional problem that generally NHibernate sessions are short - especially in web apps or web services. So when do you reset your fetching strategy. What if one portion of your code uses different customer properties than another? The adaptive fetcher needs to do a lot of analysis of your post-fetch code paths or monitor the behaviour of the application as it executes. As it stands, NHibernate has a lot of options besides adaptive queries, which I believe are better including projections (since you as a programmer know the data you need) and lazy-loading of properties - both collections and objects. There are probably others. We're talking about saving milliseconds on DB queries when a round-trip to the DB is at least an order of magnitude greater. I personally feel that adaptive queries would require a lot of work for little gain. I call YAGNI.

This is probably my fault, but his first question tells me he doesn't quite understand what I'm trying to propose. It doesn't matter if customers are important or unimportant. Once the both code paths are hit and the strategy fully adapts, it is adapted and it will fetch a superset of fields that are required for that query in the context it is called. There's no reason to ever reset that strategy unless the code changes. 

Context is another important aspect of the adaptive queries, and I'm not sure how I'd implement it. At the moment I'm thinking that something along the line of scopes (nested or single level) so for each scope/query combination there would be a strategy. That's the answer to his second issue. The only analysis it needs to do is it needs to pay attention to what properties are hit on the entities it fetched by proxying that entity. That's it. No instrumentation, parsing, or any other crazy stuff.

using (Query.Scope("Print Customer Stuff"))
{
  customers = LoadCustomers();
  ...
}

As for the YAGNI assertion, I understand why you'd call YAGNI on the 2-12ms standard savings I showed in my tests, but You Already Do Need It at times (we use projections for just this) it's just that this would be automatic and require less maintenance and you wouldn't have to choose to do it. It would be free savings and require less manual optimization. If someone further down your chain decides they need to log customer.Name, you don't have to climb back up, find the original query and add it there. With projections at least you'd know you'd need to add it, but you'd have to change the query and change your DTO (anonymous types will help with this... I guess).

My point is, You don't need an inversion of control container, it's just easier. You don't need auto mocking containers for tests, it's just easier and you don't have to change your code when you change your constructor. You don't even need Mocks, you could write those by hand too. Well, with adaptive queries you don't have to change your code when you decide to access another field... or even another collection....

You also have to look further than field adapting. There's the potential to adapt collection initialization as well. No more select N+1's for those devs that don't pay attention, they'd just go away like magic. And yes, I realize hand optimized queries will generally prevail, but adaptively optimized queries will be free.

Today, Jacob had another great idea for a use of adaptive queries. We often do this in our MonoRail actions:

public void ShowCourseEnrollments([EntityParameter] User user)
{
  foreach (CourseEnrollment ce in user.CourseEnrollments)
  {
    // Do something with ce.Course.Number
  }
}

This probably requires a bit of explanation. Basically, when someone goes to the url /Controller/ShowCourseEnrollments.rails?user=1, MR's databinding (and our EntityParameter binder) will do an NHibernate session.Load<User>(1). At this time, the db hasn't been hit. As soon as we start enumerating user.CourseEnrollments, we're select N+1ing. Furthermore, we could potentially be doing another fetch for ce.Course. The solution to this is to either change your mapping to always fetch these things (bad idea anyone?) or to do something like this:

user = UserDao.FetchUserWithEnrollments(user.Id);

Well, what if adaptive queries kicked in at the databind, and instead of adding that method to your dao, you just got what you needed? Sure, YAGNI, but You Are Gonna Want It... if I or someone ever implements it.

Ayende posted a great comment with some questions about adaptive fetching. Here are his questions and my responses:

Let us assume this:
customers = LoadCustomers();
for customer in customers:
if customer.IsImportant:
print customer.Birthday
else:
print customer.CurrentCharge
What would the adaptive fetching do in this case?
Assume that you have started with mostly unimportant customers and then moved to important customers?
The amount of queries that would be generated is prohibitive.

In this scenario, adaptive fetching would generate a query that pulled IsImportant and CurrentCharge for unimportant customers. As soon as a single Important customer ran through this code, the query would change to fetch IsImportant, CurrentCharge and Birthday. It would also immediately lazy load *all* missing properties from the original query for all customers originally queried, and continue to track accessed properties. That's only one additional query for each differing codepath, and that's only the first time its hit. From that point on, until the query was reset (version upgrade, app restart if it's not persisted, manually, etc), then you would have all you needed for all codepaths.

Another problem that you have here is that you do a query like:
"select u.Name, u.Email from Customer" and the query actually returns you Name,Email, Address, Photo.
That violates the law of least surprise fairly drastically.

I don't see adaptive queries having that syntax. I'm thinking more along the lines of "select ??? from customer"  or "select what  i need from customer", etc. Specifying what you're looking for puts you right back into projections. If you want to be specific, use projections.

Finally, I think that a much easier solution than trying to eek a few more milliseconds from a DB query is not to go to the DB at all. Utilize NH's caching abilities, and you'll get a significant performance benefit for little cost.

I agree that most of the time you'd get more benefit from caching than squeezing some time out of the db. I just think it gets us one step closer to making mapping between objects and the db more friendly to both sides of the map, and gives developers a tool that works automatically "for free". I'm sure it'd also make DBAs happier that their devs aren't just doing select * all the time.

Now, if you still want adaptive fetching, the best way to get it is to help build the HQL Parser, which would generate a human workable AST.

Agreed. I guess if we work on rewriting chunks of NHibernate the code would become manageable eh? So yeah, I'll stop blathering and try and start contributing.

I feel like I'm taking crazy pills over here. Nobody seems to see this the way I am, so I must be the crazy one, right? Let me try and explain with a few examples and pose a few specific questions that may help me understand what I'm missing. The examples are going to be somewhat contrived, but please bear with me.

Let's say we're working with the Northwind Database and for some reason one of the most often hit pages in our app is a page that displays the Freight and ShipName for a particular customer (I told you this would be contrived). Let's examine a few ways we could build the query for this.

Normal query

"from Order where CustomerId='QUICK'"

Simple. This is what we'd normally do the first time we write the app.

Normal Projection

"select o.Id, o.Freight, o.ShipName from Order o where CustomerId='QUICK'"

This is simple too. It'll give us back a list of object arrays that we can enumerate to display what we want to display.

DTO Projection

"select new OrderDto(o.Id, o.Freight, o.ShipName) from Order o where CustomerId='QUICK'"

This is a cleaner way to do a projection. Here's a good post on exactly how to do this (the key is adding an import statement to your mappings).

Partial Query

"select o(Id, Freight, ShipName) from Order o where CustomerId='QUICK'"

I made up the syntax for this. The idea is that it wouldn't use a constructor (hence no new) it'd use NHibernate's mapping to inject the property values, whether it be property setting, field setting or whatever.

What do you think of the syntax?

So let's talk about the differences between Partial Query and DTO Projection because they're the two most interesting methods to me. Projection is nice because it's implemented. You can use it today. It's also very light, it's not added to the unit of work so there's no overhead on flush, and it seems like a good fit for displaying tables of data.

On the other hand, let's pretend you've got some stupid business rule like every Tuesday the Freight on the ship named Mayflower is reported as double. Yeah, stupid rule but I don't feel like coming up with a real example. Any ways, you'd probably have something in your Order class to handle this stupid rule, but if you're projecting you'll need to apply this rule manually. If you just used a partial query, the business logic comes with you.

I should also mention that projections and partial queries differ greatly in that you cannot update with a projection. It is always read-only. This means that any time you want to update an entity, you need to load the whole thing. Even if you don't care about much or anything that's on the entity at the moment.

Here's some test data showing the average time it takes to query for 373 out of 200k Orders by CustomerId with and without covering indexes.

The decreases percentage wise is pretty significant, especially when you compare projection to normal queries. There's an improvement all around when you're able to use a covering index, but not nearly as much.

However, I find myself wondering if shaving anywhere from 1 to 12ms off a page load time is enough to justify optimization. There are a few things to consider. For one, the database isn't working as hard. Databases are expensive. If this optimization is made throughout the app, you'd probably notice significant enough savings. I definitely wouldn't suggest going through every query by hand and using projections or partial, but if it were automatic... then why not?

Is it worth the effort to bring all these features and adaptive fetching strategies to NHibernate to realize the Partial Query gains? Or are these numbers just not impressive enough?

Another thing I noticed is that projects are significantly I used dotTrace to figure out why the projection is so much faster than the partial query and it looks like the difference is mostly due to adding the entity to the session and the way things are hydrated. So it looks like Projections would still have their uses for even more aggressive optimizations.

Let's take another example from a domain I'm more familiar with--our own. For this example, we have a User, a Course, and a CourseRegistrationService. To register a user for a course you'd do something like this:

User user = userDao.FindByUsername('someuser');
Course course = courseDao.FindByCourseNumber(101);
courseRegistrationService.RegisterForCourse(user, course);

Does RegisterForCourse need to know a course's name? Or it's description? Or a user's phone number or address? Definitely not. Does NHibernate need to know them in order to hydrate you a business object you can pass around? Yep. Does it matter that NHibernate pulls too much info out of the db for just about everything you do? I think it does.

 I think it's debatable whether or not we should query for DTO's when it comes to displaying information about our business objects, but I think it's clear that you should use business objects for business logic. It's also clear that you don't always need all of the data in your involved business objects for everything you do with them.

Thoughts? When should you use a DTO? When should you use the business object?

What I see when I close my eyes and think about ORMs is an ORM that will:

• Allow me to get only what I need from the database to do my specific task. (Partial Queries)
• Figure out over time exactly what data I need (select clause) for my specific task with a little bit of guidance (from and where clause). (Adaptive Fetching Strategies)
• Allow me to specify intent for objects I pull from the database. If I know I'm not going to write them back, it shouldn't verify I didn't change anything and I shouldn't have to deal with the overhead of flushing them. (Read Only Queries)
• Track specific task performance when I want it to so that I can quickly identify queries that are called most often or that take the longest so that I have a starting point for my optimization efforts. (Yeah, this is completely unrelated and I'm sure it'd be easy to add)

Now if only I'd just shut up and start writing some code...

Oren replied to my original post about features I feel that NHibernate is missing. I think he may have misinterpreted some of my wants so I'll run through and reply in order.

Lazy Field Initialization

Obviously grabbing each field as it was accessed would be ludicrous. Lazy field initialization is simply something that needs to be there in order for partial object queries to work properly. It's the same idea as deciding whether or not to fetch a child collection. You query for it if you need it, you don't if you don't, and if you end up using the child collection when you didn't fetch it, it will get lazy loaded. I'm of the opinion that lazy loads are a smell, and if you hit them in anything but a border case, you're doing something wrong and you should be fetching what you need. 

I'm fine with specifying defaults on fields (don't load Photo unless I ask for it explicitly), but Lazy Field Initialization really would shine in partial queries.

Partial object queries

UserSummaryDetails is not User. Let's say I have a ReminderService that has a SendMailTo method that takes a User and a message. If all that method needs is that User's name and email address, why should I allow NHibernate to query for the entire user? It's just wasteful. Furthermore, it'd be rather tedious to have multiple Value Objects like UserSummaryDetails in your codebase just to support the different scenarios you want.

If you add Lazy Field Initialization and real partial object queries to the mix, you'd be able to query for a group of Users that just have email and name populated, and if you happen to need address, it'd requery for all of the user's addresses. You can even take it further and requery for all of the fields missing from the original user query. NHibernate should also probably slap you on the wrist (by logging it). I think I'm going to find a way to start logging all lazy loads so that we can investigate them.

Using objects like UserSummaryDetails eliminates a good chunk of the usefulness of the domain. That is,  you can no longer use the methods on the User object itself or any domain services that would consume that object.

So really what I want is the syntax I mentioned in my original post, which would just reuse NHibernate's field access strategy to set the appropriate fields. Any fields that were not set at that time would be flagged for lazy loading. On flush, only those fields that were set would be compared. Combine this with Adaptive Fetching Strategies and you've got a very powerful, self optimizing and flexible query mechanism that leaves you free to use your Domain as you wish.

Read Only Queries

Maybe it would make sense to give two sessions to each Dao, one of them that handled read only queries and one that handled read write queries. That's a good idea, but I still think it's something that should be built in (Hibernate does it). 

Join Qualifiers

I don't remember what the query was, but there was a reporting query I was trying to do involving multiple left joins that I just couldn't get to work with HQL. I was able to do it in SQL no problem by qualifying one of the joins, but every attempt in HQL hit a wall. Good point about having to support multiple db's. It's still something I'd like to see eventually.

First I want to admit something somewhat embarrassing: I don't know enough about databases. Until coming here, I've always had a DBA that could handle that. I've started to realize now how important it is for all developers to have a much better understanding about databases than I do. So I picked up this book. I was really looking to better understand optimization, especially knowing when to create indexes and optimizing queries. Though the book only had a few chapters on the subject, I felt like it greatly increased my understanding on the subject.

A few months ago I wrote about Adaptive Fetching Strategies. At that time I had little knowledge about covering indexes and how they improve performance. Oh well, it's just more reason something like that should be implemented. Before we even get to that however, NHibernate needs to have a few features added to it in order to be useful in scenarios where performance is important.

Remember folks, SELECT * is bad. NHibernate is for effectively always doing a SELECT * when you're asking for an object when it comes to index coverage. Yes, it specifies columns so it doesn't have all the problems associated with SELECT *, but it is still decidedly less performant than querying for only what you want. Compound this with the extra overhead the unneeded columns add during a flush, and you've got a pretty compelling argument to only query for what you need.

OK, so you've decided that you only want to query for what you need and you're using NHibernate. Well, you can do that... kind of. Not much differently than you can if you just used ADO.NET and DataSets though. NHibernate doesn't support lazy field initialization. This means that if you query for only username and email address from your user table, you don't get back a User. You get back a List of object[]'s. Arguably a list of object[]'s is less functional than a DataSet. Combine this with the fact that you're querying using HQL, which has a subset of functionality, predictability, and therefore optimizability of real SQL, and you start to see a big hole in whole ORM thing... if you want to optimize your pages. Yes, HQL does make writing queries more pretty and more in the domain, but it would be nice if it supported things like "on".

Obviously optimizing too early is evil, but we've had several pages now where we've needed to optimize, and just resorted to querying for a table of values. No longer are we querying for objects, that's just not performant enough. Especially in display scenarios where you're not making changes to anything as NHibernate lacks the ability to do read-only queries, so you end up with that (very) expensive flush unless you change your flushing strategy to be manual (awkward) or you detach your objects from the session (also awkward).

So in short, I feel NHibernate (and any ORM for that matter) needs the following features to really be optimization friendly:

• Lazy field initialization
• Querying for partial objects: select u(Username, Email) from User u
• Read-only queries that do not get flushed.
• Join qualifiers (on in T-SQL)

And yeah, I know it's open source and I could just do it myself, but I have nightmares about that codebase, and I hardly have the time to implement such large features. All I have the time to do is complain and wish :)

There's a good back and forth going on about something I've been thinking about for quite a while. Jacob Proffitt is basically claiming that DI's primary benefit is mockability in unit tests and that we should all but ditch it in favor of using TypeMock to mock our tests. Meanwhile, Ayende and Nate Kohari have been defending DI with reasons like "getting low coupling between our objects" (Ayende) and "simply put, dependency injection makes your code easier to change" (Nate).

Well, I think I'm just going to have to agree with both of them... to an extent. I agree that DI promotes loose coupling, but I disagree that TypeMock is too powerful (even though I said just that in our podcast with Hanselman, I'm allowed to change my mind). I think that DI has its place, and is not a Silver Bullet. There have been a number of times when I was refactoring a class, pulling out a method object, or moving something to a a sub service (in my attempts to adhere to the Single Responsibility Principle) that I've questioned whether or not that new class warrants all of the following:

1. Having an interface.
2. Being added to the container and being injected.
3. Writing new tests for just that, even though tests were already passing and will continue to pass for its parent object with the same coverage %.
4. Being mocked out of the original tests.

The problem with doing #4 with a framework like Rhino Mocks is that it requires you to do #3 (obviously) as well as #1 & #2 . You can't mock unless your methods are virtual or you're interfaced and that mock is injected. With TypeMock I can do #3 and #4 without worrying about adding my new class to the container. So why not add it to the container? Well, because a great deal of the time You Aren't Gonna Need It. I think once one other class takes a dependency on that service you have enough justification to control its creation in one place by adding it to the container. However, a great deal of the time, these one off objects you create to increase readability and maintainability in your code just end up either changing the way you test your objects in that you're testing more than one class at a time, or you end up with an interface and constructor argument explosion solely so that you can mock them in tests. In situations like that, Jacob is right--the only real benefit is mockability. Maybe we should consider using something like TypeMock in these situations? Just because something is powerful doesn't mean its evil. We just need to exercise more caution and use it only in times its warranted.

That said, I've never actually used TypeMock and this is all purely conjecture based on what I've read about it. I just think that we can learn something from the Ruby guys and from people with the viewpoints of Eli and Jacob, as long as everyone realizes that neither sides of the camp is producing Silver Bullets.

Jacob posted about the AutoMockingContainer several months ago. At that time we didn't really use it, it was just kind of an implementation of an idea. Well, we've finally started using it in some side projects (Resharper.TestDrive for example), and I must say... wow. It is most definitely the way to instantiate your subject under test most of the time. Why?

1. It decouples your tests from your constructors. This means that if you have multiple TestFixtures for a class and you want to add a new service to your constructor, you don't have to change a thing in your tests.
2. It simplifies your tests. Things are just cleaner when you're not having to create all your mock services to inject into your subject under test.
3. It helps reinforce good mock usage. The default mock strategy is dynamic mocks. You can override that if you want to, but most tests should (in my opinion) be written with dynamic mocks. Like Dave talks about you only really want to set actual expectations on zero or one mock at a time. Everything else should be more stub-like.

I've started to use a base class for all my tests. Let's take a look at ReSharper.TestDrive's test base class:

  public abstract class AutoMockingTests 
  {
    private MockRepository _mocks;
    private AutoMockingContainer _container;

    protected MockRepository Mocks
    {
      get { return _mocks; }
    }

    protected AutoMockingContainer Container
    {
      get { return _container; }
    }

    [SetUp]
    public void BaseSetup()
    {
      _mocks = new MockRepository();
      _container = new AutoMockingContainer(_mocks);
      _container.Initialize();
      Setup();
    }

    public abstract void Setup();

    public T Create<T>()
    {
      return _container.Create<T>();
    }

    public T Mock<T>() where T : class
    {
      return _container.Get<T>();
    }

    public void Provide<TService, TImplementation>()
    {
      _container.AddComponent(typeof(TImplementation).FullName, typeof(TService), typeof(TImplementation));
    }

    public void Provide<TService>(object instance)
    {
      _container.Kernel.AddComponentInstance(instance.GetType().FullName, typeof(TService), instance);
    }
  }

So what's a test look like with this base class? Let's borrow Dave's example.

  public class SearchPresenterTests : AutoMockingTests
  {
    private SearchPresenter _presenter;
    private SearchResultDTO _fakeResults;

    public override void Setup()
    {
      this._fakeResults = new SearchResultDTO();
      this._presenter = Create<SearchPresenter>();
    }

    [Test]
    public void Can_search_for_customers_by_number_of_orders()
    {
      using (_mocks.Record())
      {
        Expect
         .Call(Mock<ISearchService>().GetCustomersByOrderCount(42))
         .Return(this._fakeResults);
      }

      using (_mocks.Playback())
      {
        _presenter.SearchByOrderCount(42);
      }
    }

    [Test]
    public void Search_results_are_displayed_to_the_user()
    {
      using (_mocks.Record())
      {
        mockView.SearchResults = _fakeResults;
        SetupResult
         .For(Mock<ISearchService>().GetCustomersByOrderCount(42))
         .Return(_fakeResults);
      }

      using (_mocks.Playback())
      {
        presenter.SearchByOrderCount(42);
      }
    }
  }

Not bad eh? You can do some more complicated things too. Let's say all your presenters take a hub service called PresenterServices. Rather than mocking it and its child services and setting up expectations for each of the children you can just use the real one and do this:

      Provide<IPresenterService>(Create<PresenterService>());
      this._presenter = Create<SearchPresenter>();

Now you can refer to all your hub's child services with the Mock<T>() method.

Ok, so if you made it this far you probably want to check it out for yourself. Thanks to Ayende, the AMC it is now part of Rhino.Tools so you can check it out (svn co https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/trunk rhino-tools)  and build it yourself or just grab the current trunk build with all the dependencies here. Hope Oren doesn't mind me building and linking this... ;)

Writing ReSharper.TestDrive was kind of an exercise in tiny classes for me. I didn't TDD the whole thing because so much of it was just experimenting with ReSharper's mostly undocumented API and EnvDTE, though I did TDD a good portion of it after my initial spike. After I got a working prototype implemented I spent a lot of time refactoring it into tiny classes that for the most part follow the Single responsibility principle.

As this is the most code I've ever thrown out into the public at any one time and it was a bit of an experiment for me, I wanted to take this chance to ask the community to review my code. If you have the time, feel free to look over the code and tell me what you think. Likes/dislikes/hates/loves anything is fair game--feel free to rip on it. Maybe we'll get some interesting discussion out of it.

Source

Note: You must have ReSharper 3.0.1 in order for this to work.

When you're doing TDD you'll create two classes every time you need one. You'll create one for the class itself and you'll create one for the tests for that class. ReSharper makes it a little bit easier by allowing you to write your tests and then alt+enter on your class under test to create it. Unfortunately it will create the class in your test project and not your project under test. It may actually create it in that same file (I don't remember) which means you have to Move to File. Then you have to drag it over to the project under test and/or change the namespace. Pretty obnoxious for something we have to do so often.

So... I decided to write a ReSharper plugin to do just that. It'll also create tests from a class under test (just in case you cheated and created your class first). Heck, it'll even create all the folders you need to.

This current version makes a few assumptions about your structure and it's not configurable at all unless you actually hit the code. Here are the assumptions it makes:

• The tests for Project.Foo live in Project.Foo.Tests.
• Test classes have the "Tests" suffix.
• Test classes live in the same namespace as the classes under test.
• The tests for ClassFoo are in ClassFooTests.

I lied when I said it wasn't configurable at all. After you've used it for the first time it will have created ReSharper file templates that you can edit to customize what is generated when you create a test or a class under test. Just go to ReSharper>Options>Templates>FileTemplates>User Templates>TestDrive.

To install it just extract it somewhere and run install.cmd, or just copy the dll to your %APPDATA%\JetBrains\ReSharper\v3.0\vs8.0\Plugins\Resharper.TestDrive (obviously you'll need ReSharper installed).

To use it, just have the cursor somewhere within a class that doesn't have a test, or a test that doesn't have a class under test and hit Alt+Enter, select Create X Tests... or create X... and hit enter. In order for the light bulb/alt+enter to show up it will need to be able to find an associated project to the one you're in (Sample.Project.Tests<->Sample.Project).

I've got more plans for this but I wanted to get it out there to see what you all thought. Oh, and in case you're wondering, Bunker is just a really light nearly feature-free IoC container that Jacob wrote in a day for another project we're working on.

Binaries
Source