Sean Chambers asked how we work with a published branch so I figured I'd post on the topic since it's a somewhat interesting one. It's not trivial and it took us a few tries to get it where it is now, and it's still not quite right.

The first step to using a published branch is to create the branch. You can do that like this:

svn cp https://yoursvnserver/svn/project/trunk \
https://yoursvnserver/svn/project/branches/published \
-m "Branching published"

After that, we actually check out the whole branch. It's very useful to have both the pubilshed branch and the trunk on one machine. Note that this isn't necessarily trivial, and its feasibility entirely depends on your build. Having a location agnostic build is a very important thing, and this is one of the reasons.

Now we have both trunk and published branches. We almost always do all of our work in trunk and then merge over to published. Jacob actually wrote a PowerShell script to make the merges easier (from merge.ps1):

param([int]$rev, [string]$reverse)

if (!$rev)
{
  echo "Merge.ps1 REV [REVERSE]"
  return;
}
if (!$reverse)
{
  $branch = "Published"
  $url = "https://yoursvnserver/svn/project/trunk"
}
else
{
  $branch = "Trunk"
  $url = "https://yoursvnserver/svn/project/published"
}

$previous = $rev - 1
$range = $previous.ToString() + ":" + $rev

pushd $branch
svn merge -r $range $url
popd

The reason for doing most of the work in trunk is that often times any issues we run into on the published site will still be issues in the trunk. It makes sense to apply the work there first and then merge it over. The only time we patch published is when we need to apply a hack to make something work so that we can fix it the right way on the trunk later, or maybe that feature is completely different on the trunk and the fix would not apply. Of course this is dangerous and you have to be sure to remember to fix the underlying issue in the trunk before publishing from trunk again. Bug trackers help with that.

That brings us to publishing from trunk again. Merging everything from trunk into the published branch is a giant pain, and just won't work if you've applied many hacks to the published branch. I strongly advise against this. Instead, just start over:

svn rm https://yoursvnserver/svn/project/branches/published \
-m "Deleteing published branch"
svn cp https://yoursvnserver/svn/project/trunk \
https://yoursvnserver/svn/project/branches/published \
-m "Branching published"

By nuking it and recopying it you can just svn up on your published branch and you'll have everything from trunk. For whatever reason, not removing it before copying it again caused us issues. I'd recommend this two phase approach.

Other things we learned in the course of this are the pros and cons of shared "stuff". We have at least 10 gigs of course content and a few other resources that don't need to be in the separate branches like our main trunk and published. We pulled those into their own repositories and keep them in shared directories so both installs can reference it. We also have a set of common build scripts between the two. This is both a good and a bad thing. It's good because it removes some duplication and it allows us to use a separate repo for these scripts (which is handy for some TC build configurations that only need the scripts, though I guess we could just checkout the scripts directory from trunk...) but it's bad because sometimes things will get out of sync. We'll make changes to the shared scripts, and fix the trunk, but it doesn't make sense or it's prohibitive to fix the published branch. You can see in the screen shot I posted our published branch is currently failing. This is likely why. I'd probably recommend keeping all build scripts branched so that you don't run into these sort of issues.

The next thing to worry about is the database. I mentioned that we use multiple databases on the build server. What I didn't mention is that we also use multiple databases on our dev machines. We usually have three. One for trunk, one for trunk tests and one for published. The trunk tests db is imported "light" so our tests run in 6 minutes instead of 14. The trunk db is a nearly full copy of the production db so we have data when we poke around the site on our dev machines. We have a ConnectionStrings.config that is generated from the database info you pass the build script. You can do something like: msbuild Default.msbuild /p:DatabaseName=published and it'd build with the appropriate connection string.

For web applications you also have to worry about IIS. We have two web applications configured in IIS. One for trunk and one for published. This allows us to easily switch between the two by just changing our port or vdir in our url. They have multiple virtual directories underneath them that point to our various shared directories.

I think that's most of the tips I can think of right now. Let me know if you have any questions about anything.

I've mentioned before how much I like TeamCity, but I didn't really talk about how we use it. Just recently Jacob completed some work on multiple build configurations that make our life much easier. I thought I'd go over them here to give you an idea of how we handle continuous integration here.

Continuous Integration configurations

• CI - Trunk - This is the CI configuration everyone has. This watches the trunk of our SVN repository and builds whenever it changes it will also run our database migrations against the CI trunk database. It has all our built assemblies as artifacts. The artifacts take up a lot of space so we don't keep them around that long.
• CI - Published - This is just like CI - Trunk, but it watches our published branch, which is where we put everything that we're about to publish to the live site. We keep two branches so that we can make quick fixes to the published site without having to publish new features we're working on. This has its own database that is migrated.
• Nightly - Trunk - This runs daily rather than watching our source control. It migrates a database on our production database server that is a copy of our live database. It also builds and deploys the trunk to a test address on our production servers. This allows our team in Korea and our stakeholders to see changes every day in a safe environment. The Nightly is also a big part of our localization story, which I'll save for another post.

Database build configurations

• Snapshot - This and the other db build configurations probably deserve their own post with more details, but I'll do my best to explain these briefly. The Snapshot build configuration takes a point in time snapshot of the database and packages it into a zip file. The zip file becomes a TeamCity artifact that other projects can depend on.
• Nightly/CI Trunk/CI Published Baseline -

  These configurations import database snapshots into the database they refer to. The only time we need to do this is if a bad migration runs, or we want to "refresh" the data inside the database.

  It is important to note that we do not run CI - Trunk on a complete snapshot of the live database. When we do, it greatly increases the build time because our integration tests run significantly slower in a real database. Instead, we import a "light" database which contains all of the tables, but only the data from our static tables. The users, records, and anything else that grows as we get more and more users are just left empty. This means that we have zero sample data for these things during our integration tests, so we rely heavily on Fluent Fixtures to set up sample data.

  The other two databases do run nearly complete copies of the live database (we exclude log tables basically), so we still get to test our migrations and our site on real data.

Utility build configurations

• StatSVN - Runs StatSVN on our codebase. Somewhat useful source statistics. Mostly use it to see growth and churn.
• Duplicate Finder - Haven't really done this much to be honest, but TeamCity has build configurations whose sole purpose is finding duplicate code.

I'm completely enamored with this set of configurations. It makes so many things so painless. We still have room for improvement, managing all of the configuration differences in the sites is difficult. We also lack a one click live publish ability. We still follow a manual script for that, which is error prone and dangerous.

One of the first things we did when we got a dedicated development server was set up a continuous integration server. The natural choice at the time was CruiseControl.NET, so Jacob set off to get CC.NET installed and configured. I can't speak to that experience first hand, but I know it wasn't a fun one. Lots of XML hell and other non-fun issues. The end result wasn't too bad--it'd email us when builds failed with a log, and we could review logs on our CC.NET site. There was really nothing WOW about it--it got the job done. It was frustrating at times because the logs it spat out were difficult to visually parse; it was basically just a verbose msbuild log with a failure mixed in somewhere... and that was if we were lucky enough to get a text log and not have to filter through the build xml log. I'm sure there are some things we could have done to filter that log down, apply xslt to our build.xml to make it more readable, or configure things to be nicer, but I'm also sure it would have involved hard work and more XML hell.

Enter the free Professional Edition of TeamCity 3.0. We'd tried out TC briefly in the past and it seriously lacked in the .NET support arena. Now they tout support for msbuild (2.0 and 3.5), sln files from 2003-2008, and even a nifty .NET duplicate code finder. They also have some pretty slick support for NUnit which I'll get to in a moment.

I decided to give it a shot again, so I set up the TC server on our Ubuntu VM. Why there? Well that's where most of our server tools are: SVN, Mingle, FogBugz and now TC. But wait, aren't we a .NET shop? Yep. The build doesn't happen on our Ubuntu VM. See, TC is set up to have a server + build agents. A build agent is basically a machine that will check out code, do the build when the server tells it to, and report back results and artifacts to the build server. This way you can have multiple Build Agents--different Operating Systems, system configurations, physical locations, etc.

Installing both the Server and the Agent were relatively straight forward. There were a few annoyances with the Agent's configuration (like I couldn't get it to use a network drive as a working directory and it took some tweaking to get things to work with our directory structure,) but after a while I got things up and running.

Setting up a build configuration in Team City is an absolute breeze. You basically have a 7 step Q&A about your configuration:

1. General Settings - Name, Description, a few options and the build number format (we use our SVN revision #)
2. Version Control Settings - You can set up multiple repos to check out with a number of tools (SVN, CVS, Perforce, StarTeam, ClearCase). I'm sure it wouldn't be too difficult to write plugins for Mecurial or Git. You can also tell it where to check out relative to your working directory and whether or not you want it to check out to the agent or on the server and then have the agent download it.
3. Build Runner - Mainly .NET and Java stuff. No Rake yet, but it supports command line so you could just use that. With MSBuild you just specify your MSBuild file, the target you want to run, and any command line args you need. It's here that you specify your build artifacts as well. Build artifacts are kept along side your completed builds and easily accessible from the TC site.
4. Build Triggering - You can trigger based on VCS changes, schedule, other build configurations completing, or automatic retry.
5. Artifact dependencies - This makes it really easy to say you want Build A to use Build B's artifacts. For example, an installer configuration that will build an installer based on the last successful CI build.
6. Properties and environment variables - Self explanatory.
7. Agent requirements - Not all your agents can run all your build configurations necessarily. This will let you set requirements. You can set these based on environment variables, machine names, etc.

Once you get things set up, you get to see the really cool stuff. TeamCity is just packed full of nerdgasm features that just work out of the box. No need to scour the web for plugins, deal with XML, or anything like that. Things just work. For example, you get full status of the build while it is happening--you can see what MSBuild step the build is on, you can see the important messages in the Build Log (it filters them quite intelligently), you can even see what threads/processes are running. If you use their NUnit runner you can even see how many tests have passed/failed/been ignored.

That's not all you can see with their test runner though. You also get a full history of each test--how long each test took each time it was run, and if one fails you can see in what build it first failed, if it has been fixed yet, and you can even click to open it in your IDE if you have the Visual Studio plugin installed. The Tests tab even orders your tests by Duration so you have an idea which tests you may need to optimize.

Then there's the Statistics tab. This tab is a one stop shop for build health. You can see how long the build takes, how often it succeeds, how many tests you have, how many fail and even how big the artifacts are. You can see our build #s jump when we switched from a counter to SVN revision.

If there aren't enough features for you, don't worry, like with most JetBrains products you can extend them to your hearts content. That is if you can stomach the lack of API documentation (surprisingly, normal usage documentation is pretty good). One of our guys is working on a Campfire Notification plugin at the moment so we can get better build notifications in our Campfire. That's another post though.

All in all, we're very happy with TeamCity. It's just as free as CC.NET for a team of our size, it's much much easier to set up, and it has way more features. How could you go wrong? I'm sure its only a matter of time before we start to see rake runners, MBUnit test runners, and many other things to make TC even better. JetBrains has a winner here.