Lots of people seem to have been inspired by Tim Bray's Doing it Wrong post just like I was. There's a great response on rc3.org titled "Do you want to be in the software business?" that expresses a lot of the same issues I raised in The Ruby on Rails CMS Dilemma.

The gist of the post is that you shouldn't commission custom development unless you're going to be "in the software business". If your custom software as intellectual property that is a reason for people to invest in your company, then by all means, build it custom. If it's not, then make due with whatever existing product best fits your needs. Just because you can, doesn't mean you should.

Tim Bray's post on how enterprise systems are "Doing it Wrong" just triggered a realization for me. This whole agile vs. whatever debate has been a kind of red herring for a much larger argument about how businesses themselves ought to operate.

I've seen large companies adopt agile methodologies, and they don't magically transform as a result. Something of the earlier culture persists, and the tendency to overplan, overbuild and generally waste tons and tons of money continues.

The reason is that this whole thing isn't about the methodologies. It's not about the day-to-day operating procedures. It's about the culture of control and hubris inherent in the corporate structure. Big companies, especially successful ones, think they know what's right, and they think their success stands as proof of that skill. They underestimate the role of sheer luck in their past successes, and overestimate their ability to predict the future. So software teams in the enterprise make the mistake of believing that:

1. They know (or can know, with enough requirements gathering) exactly the nature of the problem their software should solve, and
2. The nature of that problem won't change over time. Or, it won't change fast enough to matter during the course of development to matter. Or, it will change but they'll be capable of recognizing and adjusting accordingly.

The world is much more complex and unpredictable than this, and these beliefs have become the downfall of many projects. The economist-philosopher F.A. Hayek explained the difficulty of central planning (he's writing about the economics of the entire economy, but I think this reasoning also applies within an individual organization, especially in enterprise software):

If we possess all the relevant information, if we can start out from a given system of preferences, and if we command complete knowledge of available means, the problem which remains is purely one of logic...

This, however, is emphatically not the economic problem which society faces... the "data" from which the economic calculus starts are never for the whole society "given" to a single mind which could work out the implications and can never be so given.

-- from the article The Use of Knowledge in Society

It's impossible for a single mind (or even a small group of minds) to encapsulate all the knowledge of complex enterprise operations. So a single software project should never even hope to be a perfect solution. Because of the fundamental information problem, even the requirements document could never be a perfect design for a single point in time. I believe this is the explanation for Galls Law.

Hayek goes on to explain that "emergent" order is superior (i.e. more efficient) to the order that central planners try to impose from above. We in western capitalist countries seem to know this in our bones in the post-communist era, but Hayek's analysis is particularly conclusive.

Before we turn too sharply into classical economic thought, and before I start re-reading Adam Smith and Karl Marx, let me bring this back on topic.

When you view software development through this Hayekian lens, the benefits of agile methodologies really become apparent. By avoiding upfront design, the software is allowed to emerge from authentic, real-world interactions, the same way economic order emerges in capitalist economies. By shortening the feedback loop between conception, construction, testing and usage, you're forced to start relying on empirical observations rather than predictions. By using dynamic languages and DSLs, you avoid the "information problem" (erg, more economics jargon) in translating human concepts into computing concepts.

So agile programming ideas make sense as potential solution in this whole enterprise software debacle. And Bray speaks of "bridging the gap" between the elephantoid enterprise software world and the grasshopper-like "web world". He closes with this word of advice:

[Successful systems like Twitter] simply cannot be built if you start with large formal specifications and fixed-price contracts and change-control procedures and so on. So if your enterprise wants the sort of outcomes we’re seeing on the Web (and a lot more should), you’re going to have to adopt some of the cultures and technologies that got them built.

He's right in the observation, but I think he's under-prescribing on the solution. It's going to take a lot more than adopting agile for in-house enterprise development at Citibank to start performing on par with Google's programming teams, for example. It's the corporate culture, not the development culture, that's to blame for these problems. It's the hubris of the organization that causes overspecification of documents, not the hubris of the software developer. In practical terms, it's the CEO, not the CTO that's got to lead the type of gap-bridging that Bray is speaking of.

Large companies (or "Enterprises", I guess. I can't seem to use that term without parentheses or quotes... it's like the term itself is an ironic joke) won't start making better use of IT until they realize that great software is a process not a project. And the process has to start outside the IT department if it has any chance of being successful.

I've just completed my Rails Rumble contest entry for this year. I took a very focused project, and built it as a single solo developer. I'm extremely pleased with the way it turned out.

The site was inspired by a recent post over at kottke.org about "The Steve Ward" diet (see Kottke's source article from Philip Greenspun, it's a good read... and not actually about dieting).

After reading about this simple diet strategy (it's nothing more than a graph paper of your daily weight), I decided to try it for myself; it works. I've lost 8 pounds in about 3 weeks. My success is mostly due to more frequent and longer trips to the gym--the motivation for that has come directly from the accurate daily feedback you get from this plan.

The web site is a slightly more sophisticated version of the paper-and-pen version Steve Ward describes.

Check it out:

Introductory Screencast
The site itself - The Line Diet
My Rails Rumble Team page

The Rails Rumble contest opens for public voting on August 27th.

This is one of my own personal Git FAQs. Here's the scenario: You've created a topic branch of new code, and done all the work required to complete the feature. But before you can merge it back into the main tree, you get assigned to some new work, and the branch sits dormant for several days. Now the time has come to merge that feature back in, but it's been so long, you forgot exactly what the code included. Does it conflict with anything new? Was part of the problem solved with some later code? Here's how you find out:


git co development # The destination branch
git diff HEAD...topic-branch


You've probably guessed that the magic there is the .... In English, that symbol means show the difference between the common ancestor commit and HEAD.... in other words, only show what changes in the topic branch while it was distinct from the branch it's being merged into.

Ruby on Rails is an excellent framework for building web applications. Perhaps the best. But it's not currently very well suited to what I call web sites.

The difference is simple. In a web site, the unique business value comes from the content creators (authors, bloggers, photographers, etc). In a web application, the business value comes directly from the programmers. Twitter, Google, Basecamp and eBay are web applications. CBSSports.com, KentuckyDerby.com, corporate brand sites and original news sources are all web sites.

For these projects where it's the content that matters, there's no reason to spend time hand-coding models, controllers, views and test code for things called Pages, Articles, BlogPosts, CalendarEntries, and all the other unique types of content. You're just re-inventing the wheel if you do, because there are numerous existing commercial and open source CMS tools that can do the job cheaper and better than you.

Unfortunately, I find that one of the most common scenarios is a hybrid web site that needs an attached web application. A corporate web site that needs an attached scheduling tool. A news site that needs an interactive, custom-searchable database. A sports site which aggregates content from an old NNTP-based data source.

Depending on the specifics, you usually have 3 different approaches to this problem:

1. Two side-by-side applications. - Pair an off-the-shelf CMS with a custom built application. Copy/paste the visual elements, and loosely wire the two together with a subdomain or some kind of web proxy to share the URL structure
2. Extend the CMS - Build a proprietary plugin or extension for the CMS to implement the application features.
3. Soup-to-nuts custom application. Build a complete CMS as part of the project.

If scenario #1 is truly possible, it's the best approach. Unfortunately, it's often the case that the two systems need to share user accounts, group permissions (or worse, business logic) and data. Once you link the code and data of the two applications, you actually have a particularly "smelly" case of scenario #2.

And scenario #2 is usually a bad idea, unless you are really committed to the CMS you've chosen. This approach tightly couples your value-adding intellectual property directly to a CMS that generally has nothing to do with your business. If your CMS vendor is out of business in 5 years, you'll probably need to completely rewrite that custom extension to work in some new system. Unfortunately, this seems to be the most commonly-used approach. Radiant's extensions have this problem, and so does the Block/Portlet system in BrowserCMS.

Finally, scenario #3 is just plain wasteful. If your CMS needs are anything but trivial, you're going to spend a lot of time and money building CMS features that have nothing to do with your business enterprise. Unless you accidentally build an exceptionally good CMS tool that you then sell to others, all that money is wasted.

The reason for this problem is that Ruby on Rails is a framework for applications, not systems. The error in all of the reasoning above is that the described CMS project is not a single application, but a system of two (or more) applications.

I've seen some very attractive developments that will help to address this issue in future Rails applications. First, the inclusion of Rails Engines in 2.3 is going to lower the level of coupling for cases where the CMS can be an "engine" within the application. There's some hope that an even more flexible solution may be forthcoming. These solutions help to flip scenario #2 on its head. Unfortunately, this still leaves your application largely dependent on the CMS vendor. You may have trouble customizing the CMS' features without hacking the engine (or "mounted app") code directly.

Over the past few months, I've been working to find better solutions to the issues I've raised here. I'm in the process of building an experimental engines-based CMS plugin, through which I hope to gain some active insight into what makes this problem so difficult. I'll be posting my analysis as it crystalizes. If the CMS turns out to be effective for my uses, I'll be releasing it to GitHub.

For now, I'll simply leave it at this: I believe the solution lies in finding a common "language" for an application to communicate with its plugins, and for those plugins to communicate with one another. In the case of authentication and authorization, this means a plugin needs to know how to ask about the identity and rights of the active user.

Django and Drupal, have made the framework itself answer that question (both Django and Drupal have admin interfaces and user management features somewhat baked in). I'm not convinced that's the best approach, but it seems worthwhile to consider that approach for Rails.

My CMS plugin is opting instead to look for API functions at the controller level to answer these questions (i.e. the plugin assumes you have a current_user() method in ApplicationController, but allows you to customize the name and behavior of that).

Update: Thank you for all the encouraging comments. There are sure some good people out there in the Rails community. I wanted to draw some attention to ferrisoxide's blog post: "Rails is a CMS". This thought is very much along the lines of the experimental development I'm doing. The goal for my approach is to leverage as much of the flexibility of Rails itself for it's feature set.... in that way I'm planning more of a CMS-enabler plugin than an actual out-of-the box CMS.

Update 2: I wanted to also draw attention to Aaron Bedra's comments about CAS, specifically Castronaut as an implementation pattern for scenario #1. It didn't even cross my mind when I was writing this post... and CAS seems to be one of the most under-hyped technologies out there. More on that later.

Rails' built-in Javascript view helpers are excellent in terms of convenience, but they make it difficult to follow Yahoo's wise advice that scripts at the bottom of the page make for faster loading pages than scripts at the top of the page (obviously, using inline Javascript like the Rails helpers do requires that your <%= javascript_include_tag :defaults %> be above the main yield in the template.

The solution to this problem is simple. Every time you need to call a Javascript helper method in a view, just wrap it in a <% content_for :footer do %> block, then put a <%= yield :footer > just before the </body> in your template.