Tim Bray's post on how enterprise systems are "Doing it Wrong" just triggered a realization for me. This whole agile vs. whatever debate has been a kind of red herring for a much larger argument about how businesses themselves ought to operate.

I've seen large companies adopt agile methodologies, and they don't magically transform as a result. Something of the earlier culture persists, and the tendency to overplan, overbuild and generally waste tons and tons of money continues.

The reason is that this whole thing isn't about the methodologies. It's not about the day-to-day operating procedures. It's about the culture of control and hubris inherent in the corporate structure. Big companies, especially successful ones, think they know what's right, and they think their success stands as proof of that skill. They underestimate the role of sheer luck in their past successes, and overestimate their ability to predict the future. So software teams in the enterprise make the mistake of believing that:

1. They know (or can know, with enough requirements gathering) exactly the nature of the problem their software should solve, and
2. The nature of that problem won't change over time. Or, it won't change fast enough to matter during the course of development to matter. Or, it will change but they'll be capable of recognizing and adjusting accordingly.

The world is much more complex and unpredictable than this, and these beliefs have become the downfall of many projects. The economist-philosopher F.A. Hayek explained the difficulty of central planning (he's writing about the economics of the entire economy, but I think this reasoning also applies within an individual organization, especially in enterprise software):

If we possess all the relevant information, if we can start out from a given system of preferences, and if we command complete knowledge of available means, the problem which remains is purely one of logic...

This, however, is emphatically not the economic problem which society faces... the "data" from which the economic calculus starts are never for the whole society "given" to a single mind which could work out the implications and can never be so given.

-- from the article The Use of Knowledge in Society

It's impossible for a single mind (or even a small group of minds) to encapsulate all the knowledge of complex enterprise operations. So a single software project should never even hope to be a perfect solution. Because of the fundamental information problem, even the requirements document could never be a perfect design for a single point in time. I believe this is the explanation for Galls Law.

Hayek goes on to explain that "emergent" order is superior (i.e. more efficient) to the order that central planners try to impose from above. We in western capitalist countries seem to know this in our bones in the post-communist era, but Hayek's analysis is particularly conclusive.

Before we turn too sharply into classical economic thought, and before I start re-reading Adam Smith and Karl Marx, let me bring this back on topic.

When you view software development through this Hayekian lens, the benefits of agile methodologies really become apparent. By avoiding upfront design, the software is allowed to emerge from authentic, real-world interactions, the same way economic order emerges in capitalist economies. By shortening the feedback loop between conception, construction, testing and usage, you're forced to start relying on empirical observations rather than predictions. By using dynamic languages and DSLs, you avoid the "information problem" (erg, more economics jargon) in translating human concepts into computing concepts.

So agile programming ideas make sense as potential solution in this whole enterprise software debacle. And Bray speaks of "bridging the gap" between the elephantoid enterprise software world and the grasshopper-like "web world". He closes with this word of advice:

[Successful systems like Twitter] simply cannot be built if you start with large formal specifications and fixed-price contracts and change-control procedures and so on. So if your enterprise wants the sort of outcomes we’re seeing on the Web (and a lot more should), you’re going to have to adopt some of the cultures and technologies that got them built.

He's right in the observation, but I think he's under-prescribing on the solution. It's going to take a lot more than adopting agile for in-house enterprise development at Citibank to start performing on par with Google's programming teams, for example. It's the corporate culture, not the development culture, that's to blame for these problems. It's the hubris of the organization that causes overspecification of documents, not the hubris of the software developer. In practical terms, it's the CEO, not the CTO that's got to lead the type of gap-bridging that Bray is speaking of.

Large companies (or "Enterprises", I guess. I can't seem to use that term without parentheses or quotes... it's like the term itself is an ironic joke) won't start making better use of IT until they realize that great software is a process not a project. And the process has to start outside the IT department if it has any chance of being successful.

Alan Kohler has some commentary over at Business Spectator that suggests a stunning ending for the global financial crisis; one that actually involves a massive windfall for the banks themselves.

It is now getting very interesting. The three Icelandic banks have defaulted, as has Countrywide, Lehman and Bear Stearns. AIG has been taken over by the US Government, which is counted as a part-default, and Freddie Mac and Fannie Mae are in “conservatorship”...

Ambac, MBIA, PMI, General Motors, Ford and a lot of US home builders are teetering.

If the list of defaults – full and partial – gets to nine, then a mass transfer of money will take place from unsuspecting investors around the world into the banking system. How much? Nobody knows, but it’s many trillions.

...

The distress among those who lose their money will be immense. It will be a real loss, not a theoretical paper loss. Cash will be transferred from their own bank accounts into the issuing bank, via these Cayman Islands special purpose vehicles.

The repercussions on the losers and the economies in which they live, will be unpredictable but definitely huge...

There will also be a tsunami of litigation, as dumbfounded investors try to get their money back, claiming to have been deceived by the sales people who sold them the products...

But for the banks, it’s happy days. Suddenly, when the ninth reference entity tips over, they will be flooded with capital. It’s possible they will have so much new capital, they won’t know what to do with it.