SpeakENG

First some code:

UserEntity user = New.User("bob").In(New.Company("ACME")).With(
  New.PurchaseOf("ProductA")
    .WithAcademicSlot(1).FilledBy(New.EnrollmentIn(101).ThatIsCanceled)
    .WithPlacementSlot(1).FilledBy(New.EnrollmentIn(1)));

This code actually creates and saves (with NHibernate) about 12 or so entities. We can use code like this to set up one off sample data for tests in a way that's easy to read, understand and change. There is a decent amount of magic that goes into making this work and I wanted to talk about how I did it.

First we have a FixtureContext, which is just a hub for the DaoFactory, the current Session, and three helper classes, New, Current and Existing. New is the class responsible for the beginning of the syntax you see above. Each method on New returns a Creator. There's some magic in Creator, so here's the code:

public class Creator<T> : FixtureContextAware where T: class, new()
{
  private T _creation;
  
  protected T Creation
  {
    get { return _creation; }
    private set 
    {
      if (Current.Get<T>() != value)
      {
        Current.Push(value);
      }
      _creation = value; 
    }
  }

  public Creator(IFixtureContext context) : base(context)
  {
    Creation = new T();
  }

  public Creator(IFixtureContext context, T creation) : base(context)
  {
    Creation = creation;
  }

  public static implicit operator T(Creator<T> creator)
  {
    if (creator._creation == null) throw new Exception(
      String.Format("Creation of {0} is null, it probably shouldn't be.", typeof(T)));
    creator.Current.Pop<T>();
    return creator._creation;
  }
}

The first thing is that Creator is a subclass of FixtureContextAware, which is just a helper base class that provides access to FixtureContext's children. Next there are a few references to Current which is simply a collection of stacks of entities so that Creators can refer to other entities that are being created so they don't have to be passed around. This is better explained with an example. In the beginning example you see New.EnrollmentIn(101). An enrollment requires a User to be created, so because there is a User in this creation context, we can do this:

public CourseEnrollmentCreator(IFixtureContext context, short number) : base(context)
{
  Creation.User = Current.User;
  Creation.Course = Existing.Course(number);
  Creation.StartDate = DateTime.Now;
  Creation.EndDate = DateTime.Now.AddMonths(1);

  Session.Save(Creation);
}

The next thing is that the creation itself is stored as Creation in the Creator. This can either be new'd up or can be passed in to the constructor.

The coolest part (at least in my opinion) is the implicit operator. This allows you do to do things like: UserEntity user = New.User().Foo(), where each of those methods returns a UserCreator, but at the end of all of it the Creator is implicitly cast to a UserEntity, the thing actually being created. This also serves as an excellent time to pop the entity from the Current stack.

Next we have the Existing class. This is essentially just a wrapper for your Daos/Repositories so you can fetch things that are already in your database (like Existing.Course(number) or Existing.User("bob")).

With this simple framework in place, the next step is to start writing your domain specific Creators. Here's an example:

public class PurchaseCreator : Creator<PurchasedProductEntity>
{
  protected PurchaseCreator(IFixtureContext context, PurchasedProductEntity creation) : base(context, creation)
  {
  }

  public PurchaseCreator(IFixtureContext context, string productName) : base(context)
  {
    Creation.Product = New.Product(productName);
    // Initialize Purchase...
  }

  public PurchaseCreatorWithSlot WithAcademicSlot(int credits)
  {
    // Create and add slot...
   
    return new PurchaseCreatorWithSlot(Context, this, slot);
  }

  public PurchaseCreatorWithSlot WithPlacementSlot(int credits)
  {
    // Create and add slot...

    return new PurchaseCreatorWithSlot(Context, this, slot);
  }

  public PurchaseCreator WithSessions(int credits)
  {
    // Create and add sessions...

    return this;
  }
}

public class PurchaseCreatorWithSlot : PurchaseCreator
{
  private readonly PurchasedCourseEnrollmentSlotEntity _slot;

  public PurchaseCreatorWithSlot(IFixtureContext context, PurchasedProductEntity creation, PurchasedCourseEnrollmentSlotEntity slot) : base(context, creation)
  {
    _slot = slot;
  }

  public PurchaseCreatorWithSlot FilledBy(CourseEnrollmentEntity enrollment)
  {
    // Fill slot...
  }
}

Then we add a PurchaseOf(string productName) method to New that will create and return a new PurchaseCreator. We can also add a PurchaseOf(ProductEntity product) and pass a New.Product("Product") in instead. You can make it as granular or magic as you want. Also, Notice that WithAcademicSlot and WithPlacementSlot return a subclass of PurchaseCreator that adds another method. Using techniques like this you can make some very verbose and context sensitive fixtures.

We also make our HibernateTests base class FixtureContextAware so that we can use nice syntax in our tests.

Here is the source for the basic framework. Let me know what you think. 

I just had to try the typing speed test that Mitch and Darren tried. Here are my results:

Wanting to try and be a bit more accurate since Darren said it's important, I tried it again (and yes, it was different text, so I didn't have any additional advantage other than the burning in my forearms... which isn't quite an advantage):

I just can't seem to break 100 WPM and I don't see myself ever getting over 98% accuracy, ah well.

OK, so it's not 100% free as you'll need a few drywall anchors (if you're not on a stud) and screws, but you probably have those laying around, and other than that it's free. When you're done, you'll be able to move the monitor up and down, tilt it up and down, and rotate it, just like you can on its stand.

I started taking it apart to see what I could do with it and I didn't start taking pictures until I was almost done, so you're going to have to use your imagination a bit on some of the pics and some of the steps may be off a bit. It looks like a lot of steps, but they're tiny and somewhat detailed... all in all it's pretty easy. Any ways, here you go:

1. Pull off the monitor from the stand by pushing the black button and rotating it off.
   
2. Unscrew the four screws that mount the connecting plate to the stand and pull the connecting plate off.
   
3. This may be optional, but it's easy: unscrew the two little screws on the gray plastic circle and pull it off.
   
4. This is also probably optional: unscrew the two screws holding the plastic hinge cover on and pull off the cover.
5. Pull off the bottom black plastic piece from the track on the stand.
   
6. Use all of your might to push the mount down to the bottom of the track until it locks.
7. Pull off the rest of the black track cover (this is a little tricky, and I broke some of the tabs doing it, but who cares, right?) Basically you have to pull the sides out from each other and go all the way up to the top, then slide the middle part out.
8. Push the button so that the mount slides (very quickly, watch your hands) to the top.
9. Unscrew the three screws holding the long metal piece down to the track assembly and pull it out. Things will fall out, don't worry that's fine.
   
10. Unscrew the four tiny screws holding the track assembly to the stand. These screws were a pain because they're in there pretty tight and they're so small. I stripped one of them and had to drill it out.
11. Unscrew the four screws holding the rear bracket to the two track rails.
    
12. Rotate the rails up and pop the metal coil out of the plastic thing.
    
13. Mount the rear bracket using the incredibly convenient four screw holes to the wall. I used some 2"x1/4" plastic drywall anchors and 1.5" screws. If you have a stud there, all the better, but it may be tricky getting all 4 screws into the stud.
    
14. Pop the metal coil into the plastic thing at the top of the bracket. Check to make sure it got in there straight or you may have some annoying problems.
    
15. Replace the four screws to secure the tracks to the bracket. In order to do this I needed to hold the mount a few inches down from the top and have my girlfriend screw the screws into the rails. That last sentence may get me some unexpected traffic from google.
    
16. Purely optional: Replace the two plastic covers and the four screws.
17. Replace the mounting plate and secure it with all four screws.
18. Play with the plate/mounting a bit to make sure its secure before putting the monitor on it.
19. Put the monitor onto the mounting plate.
    
20. Hook up your cables and enjoy--if you're feeling adventurous, feel free to drill some holes and fish those cables through the wall :)

Don't ever put a Photo directly in your User entity (or any entity commonly used for anything but displaying a photo). It's not only arguably bad from a normalization standpoint, but NHibernate's flushes will absolutely kill you. Don't say I didn't warn you.

James Kovacs replied to one of my many NHibernate Optimization Ramblings:

In the first example, how do you know that most customers are unimportant until you fetch them from the database? You have an additional problem that generally NHibernate sessions are short - especially in web apps or web services. So when do you reset your fetching strategy. What if one portion of your code uses different customer properties than another? The adaptive fetcher needs to do a lot of analysis of your post-fetch code paths or monitor the behaviour of the application as it executes. As it stands, NHibernate has a lot of options besides adaptive queries, which I believe are better including projections (since you as a programmer know the data you need) and lazy-loading of properties - both collections and objects. There are probably others. We're talking about saving milliseconds on DB queries when a round-trip to the DB is at least an order of magnitude greater. I personally feel that adaptive queries would require a lot of work for little gain. I call YAGNI.

This is probably my fault, but his first question tells me he doesn't quite understand what I'm trying to propose. It doesn't matter if customers are important or unimportant. Once the both code paths are hit and the strategy fully adapts, it is adapted and it will fetch a superset of fields that are required for that query in the context it is called. There's no reason to ever reset that strategy unless the code changes. 

Context is another important aspect of the adaptive queries, and I'm not sure how I'd implement it. At the moment I'm thinking that something along the line of scopes (nested or single level) so for each scope/query combination there would be a strategy. That's the answer to his second issue. The only analysis it needs to do is it needs to pay attention to what properties are hit on the entities it fetched by proxying that entity. That's it. No instrumentation, parsing, or any other crazy stuff.

using (Query.Scope("Print Customer Stuff"))
{
  customers = LoadCustomers();
  ...
}

As for the YAGNI assertion, I understand why you'd call YAGNI on the 2-12ms standard savings I showed in my tests, but You Already Do Need It at times (we use projections for just this) it's just that this would be automatic and require less maintenance and you wouldn't have to choose to do it. It would be free savings and require less manual optimization. If someone further down your chain decides they need to log customer.Name, you don't have to climb back up, find the original query and add it there. With projections at least you'd know you'd need to add it, but you'd have to change the query and change your DTO (anonymous types will help with this... I guess).

My point is, You don't need an inversion of control container, it's just easier. You don't need auto mocking containers for tests, it's just easier and you don't have to change your code when you change your constructor. You don't even need Mocks, you could write those by hand too. Well, with adaptive queries you don't have to change your code when you decide to access another field... or even another collection....

You also have to look further than field adapting. There's the potential to adapt collection initialization as well. No more select N+1's for those devs that don't pay attention, they'd just go away like magic. And yes, I realize hand optimized queries will generally prevail, but adaptively optimized queries will be free.

Today, Jacob had another great idea for a use of adaptive queries. We often do this in our MonoRail actions:

public void ShowCourseEnrollments([EntityParameter] User user)
{
  foreach (CourseEnrollment ce in user.CourseEnrollments)
  {
    // Do something with ce.Course.Number
  }
}

This probably requires a bit of explanation. Basically, when someone goes to the url /Controller/ShowCourseEnrollments.rails?user=1, MR's databinding (and our EntityParameter binder) will do an NHibernate session.Load<User>(1). At this time, the db hasn't been hit. As soon as we start enumerating user.CourseEnrollments, we're select N+1ing. Furthermore, we could potentially be doing another fetch for ce.Course. The solution to this is to either change your mapping to always fetch these things (bad idea anyone?) or to do something like this:

user = UserDao.FetchUserWithEnrollments(user.Id);

Well, what if adaptive queries kicked in at the databind, and instead of adding that method to your dao, you just got what you needed? Sure, YAGNI, but You Are Gonna Want It... if I or someone ever implements it.

Ayende posted a great comment with some questions about adaptive fetching. Here are his questions and my responses:

Let us assume this:
customers = LoadCustomers();
for customer in customers:
if customer.IsImportant:
print customer.Birthday
else:
print customer.CurrentCharge
What would the adaptive fetching do in this case?
Assume that you have started with mostly unimportant customers and then moved to important customers?
The amount of queries that would be generated is prohibitive.

In this scenario, adaptive fetching would generate a query that pulled IsImportant and CurrentCharge for unimportant customers. As soon as a single Important customer ran through this code, the query would change to fetch IsImportant, CurrentCharge and Birthday. It would also immediately lazy load *all* missing properties from the original query for all customers originally queried, and continue to track accessed properties. That's only one additional query for each differing codepath, and that's only the first time its hit. From that point on, until the query was reset (version upgrade, app restart if it's not persisted, manually, etc), then you would have all you needed for all codepaths.

Another problem that you have here is that you do a query like:
"select u.Name, u.Email from Customer" and the query actually returns you Name,Email, Address, Photo.
That violates the law of least surprise fairly drastically.

I don't see adaptive queries having that syntax. I'm thinking more along the lines of "select ??? from customer"  or "select what  i need from customer", etc. Specifying what you're looking for puts you right back into projections. If you want to be specific, use projections.

Finally, I think that a much easier solution than trying to eek a few more milliseconds from a DB query is not to go to the DB at all. Utilize NH's caching abilities, and you'll get a significant performance benefit for little cost.

I agree that most of the time you'd get more benefit from caching than squeezing some time out of the db. I just think it gets us one step closer to making mapping between objects and the db more friendly to both sides of the map, and gives developers a tool that works automatically "for free". I'm sure it'd also make DBAs happier that their devs aren't just doing select * all the time.

Now, if you still want adaptive fetching, the best way to get it is to help build the HQL Parser, which would generate a human workable AST.

Agreed. I guess if we work on rewriting chunks of NHibernate the code would become manageable eh? So yeah, I'll stop blathering and try and start contributing.

I feel like I'm taking crazy pills over here. Nobody seems to see this the way I am, so I must be the crazy one, right? Let me try and explain with a few examples and pose a few specific questions that may help me understand what I'm missing. The examples are going to be somewhat contrived, but please bear with me.

Let's say we're working with the Northwind Database and for some reason one of the most often hit pages in our app is a page that displays the Freight and ShipName for a particular customer (I told you this would be contrived). Let's examine a few ways we could build the query for this.

Normal query

"from Order where CustomerId='QUICK'"

Simple. This is what we'd normally do the first time we write the app.

Normal Projection

"select o.Id, o.Freight, o.ShipName from Order o where CustomerId='QUICK'"

This is simple too. It'll give us back a list of object arrays that we can enumerate to display what we want to display.

DTO Projection

"select new OrderDto(o.Id, o.Freight, o.ShipName) from Order o where CustomerId='QUICK'"

This is a cleaner way to do a projection. Here's a good post on exactly how to do this (the key is adding an import statement to your mappings).

Partial Query

"select o(Id, Freight, ShipName) from Order o where CustomerId='QUICK'"

I made up the syntax for this. The idea is that it wouldn't use a constructor (hence no new) it'd use NHibernate's mapping to inject the property values, whether it be property setting, field setting or whatever.

What do you think of the syntax?

So let's talk about the differences between Partial Query and DTO Projection because they're the two most interesting methods to me. Projection is nice because it's implemented. You can use it today. It's also very light, it's not added to the unit of work so there's no overhead on flush, and it seems like a good fit for displaying tables of data.

On the other hand, let's pretend you've got some stupid business rule like every Tuesday the Freight on the ship named Mayflower is reported as double. Yeah, stupid rule but I don't feel like coming up with a real example. Any ways, you'd probably have something in your Order class to handle this stupid rule, but if you're projecting you'll need to apply this rule manually. If you just used a partial query, the business logic comes with you.

I should also mention that projections and partial queries differ greatly in that you cannot update with a projection. It is always read-only. This means that any time you want to update an entity, you need to load the whole thing. Even if you don't care about much or anything that's on the entity at the moment.

Here's some test data showing the average time it takes to query for 373 out of 200k Orders by CustomerId with and without covering indexes.

The decreases percentage wise is pretty significant, especially when you compare projection to normal queries. There's an improvement all around when you're able to use a covering index, but not nearly as much.

However, I find myself wondering if shaving anywhere from 1 to 12ms off a page load time is enough to justify optimization. There are a few things to consider. For one, the database isn't working as hard. Databases are expensive. If this optimization is made throughout the app, you'd probably notice significant enough savings. I definitely wouldn't suggest going through every query by hand and using projections or partial, but if it were automatic... then why not?

Is it worth the effort to bring all these features and adaptive fetching strategies to NHibernate to realize the Partial Query gains? Or are these numbers just not impressive enough?

Another thing I noticed is that projects are significantly I used dotTrace to figure out why the projection is so much faster than the partial query and it looks like the difference is mostly due to adding the entity to the session and the way things are hydrated. So it looks like Projections would still have their uses for even more aggressive optimizations.

Let's take another example from a domain I'm more familiar with--our own. For this example, we have a User, a Course, and a CourseRegistrationService. To register a user for a course you'd do something like this:

User user = userDao.FindByUsername('someuser');
Course course = courseDao.FindByCourseNumber(101);
courseRegistrationService.RegisterForCourse(user, course);

Does RegisterForCourse need to know a course's name? Or it's description? Or a user's phone number or address? Definitely not. Does NHibernate need to know them in order to hydrate you a business object you can pass around? Yep. Does it matter that NHibernate pulls too much info out of the db for just about everything you do? I think it does.

 I think it's debatable whether or not we should query for DTO's when it comes to displaying information about our business objects, but I think it's clear that you should use business objects for business logic. It's also clear that you don't always need all of the data in your involved business objects for everything you do with them.

Thoughts? When should you use a DTO? When should you use the business object?

What I see when I close my eyes and think about ORMs is an ORM that will:

• Allow me to get only what I need from the database to do my specific task. (Partial Queries)
• Figure out over time exactly what data I need (select clause) for my specific task with a little bit of guidance (from and where clause). (Adaptive Fetching Strategies)
• Allow me to specify intent for objects I pull from the database. If I know I'm not going to write them back, it shouldn't verify I didn't change anything and I shouldn't have to deal with the overhead of flushing them. (Read Only Queries)
• Track specific task performance when I want it to so that I can quickly identify queries that are called most often or that take the longest so that I have a starting point for my optimization efforts. (Yeah, this is completely unrelated and I'm sure it'd be easy to add)

Now if only I'd just shut up and start writing some code...

Oren replied to my original post about features I feel that NHibernate is missing. I think he may have misinterpreted some of my wants so I'll run through and reply in order.

Lazy Field Initialization

Obviously grabbing each field as it was accessed would be ludicrous. Lazy field initialization is simply something that needs to be there in order for partial object queries to work properly. It's the same idea as deciding whether or not to fetch a child collection. You query for it if you need it, you don't if you don't, and if you end up using the child collection when you didn't fetch it, it will get lazy loaded. I'm of the opinion that lazy loads are a smell, and if you hit them in anything but a border case, you're doing something wrong and you should be fetching what you need. 

I'm fine with specifying defaults on fields (don't load Photo unless I ask for it explicitly), but Lazy Field Initialization really would shine in partial queries.

Partial object queries

UserSummaryDetails is not User. Let's say I have a ReminderService that has a SendMailTo method that takes a User and a message. If all that method needs is that User's name and email address, why should I allow NHibernate to query for the entire user? It's just wasteful. Furthermore, it'd be rather tedious to have multiple Value Objects like UserSummaryDetails in your codebase just to support the different scenarios you want.

If you add Lazy Field Initialization and real partial object queries to the mix, you'd be able to query for a group of Users that just have email and name populated, and if you happen to need address, it'd requery for all of the user's addresses. You can even take it further and requery for all of the fields missing from the original user query. NHibernate should also probably slap you on the wrist (by logging it). I think I'm going to find a way to start logging all lazy loads so that we can investigate them.

Using objects like UserSummaryDetails eliminates a good chunk of the usefulness of the domain. That is,  you can no longer use the methods on the User object itself or any domain services that would consume that object.

So really what I want is the syntax I mentioned in my original post, which would just reuse NHibernate's field access strategy to set the appropriate fields. Any fields that were not set at that time would be flagged for lazy loading. On flush, only those fields that were set would be compared. Combine this with Adaptive Fetching Strategies and you've got a very powerful, self optimizing and flexible query mechanism that leaves you free to use your Domain as you wish.

Read Only Queries

Maybe it would make sense to give two sessions to each Dao, one of them that handled read only queries and one that handled read write queries. That's a good idea, but I still think it's something that should be built in (Hibernate does it). 

Join Qualifiers

I don't remember what the query was, but there was a reporting query I was trying to do involving multiple left joins that I just couldn't get to work with HQL. I was able to do it in SQL no problem by qualifying one of the joins, but every attempt in HQL hit a wall. Good point about having to support multiple db's. It's still something I'd like to see eventually.

First I want to admit something somewhat embarrassing: I don't know enough about databases. Until coming here, I've always had a DBA that could handle that. I've started to realize now how important it is for all developers to have a much better understanding about databases than I do. So I picked up this book. I was really looking to better understand optimization, especially knowing when to create indexes and optimizing queries. Though the book only had a few chapters on the subject, I felt like it greatly increased my understanding on the subject.

A few months ago I wrote about Adaptive Fetching Strategies. At that time I had little knowledge about covering indexes and how they improve performance. Oh well, it's just more reason something like that should be implemented. Before we even get to that however, NHibernate needs to have a few features added to it in order to be useful in scenarios where performance is important.

Remember folks, SELECT * is bad. NHibernate is for effectively always doing a SELECT * when you're asking for an object when it comes to index coverage. Yes, it specifies columns so it doesn't have all the problems associated with SELECT *, but it is still decidedly less performant than querying for only what you want. Compound this with the extra overhead the unneeded columns add during a flush, and you've got a pretty compelling argument to only query for what you need.

OK, so you've decided that you only want to query for what you need and you're using NHibernate. Well, you can do that... kind of. Not much differently than you can if you just used ADO.NET and DataSets though. NHibernate doesn't support lazy field initialization. This means that if you query for only username and email address from your user table, you don't get back a User. You get back a List of object[]'s. Arguably a list of object[]'s is less functional than a DataSet. Combine this with the fact that you're querying using HQL, which has a subset of functionality, predictability, and therefore optimizability of real SQL, and you start to see a big hole in whole ORM thing... if you want to optimize your pages. Yes, HQL does make writing queries more pretty and more in the domain, but it would be nice if it supported things like "on".

Obviously optimizing too early is evil, but we've had several pages now where we've needed to optimize, and just resorted to querying for a table of values. No longer are we querying for objects, that's just not performant enough. Especially in display scenarios where you're not making changes to anything as NHibernate lacks the ability to do read-only queries, so you end up with that (very) expensive flush unless you change your flushing strategy to be manual (awkward) or you detach your objects from the session (also awkward).

So in short, I feel NHibernate (and any ORM for that matter) needs the following features to really be optimization friendly:

• Lazy field initialization
• Querying for partial objects: select u(Username, Email) from User u
• Read-only queries that do not get flushed.
• Join qualifiers (on in T-SQL)

And yeah, I know it's open source and I could just do it myself, but I have nightmares about that codebase, and I hardly have the time to implement such large features. All I have the time to do is complain and wish :)