On Systems Architecture

The following article was originally published in the Proceedings of the 1995 Monterey Workshop on Specification-Based Software Architectures [US Naval Postgraduate School, Monterey, California, September, 1995.]

In this brief reflection, I shall comment on systems architecture and the effect of its absence, lay down some definitions, and then end with a discussion of the odd dynamic that impedes the successful architecting of most software. It is in this last section that I address what I believe to be our major architectural problem: since we attribute most architectural failure to the wrong cause, we are not making much progress in the area. Just to give you a hint of what is to come: It is my thesis that though we think of the inhibitors of successful architecture as entirely technological, they are much more likely to be economic, political and sociological.

Architecture and its absence

The architecture of a complex software system is analogous to the infrastructure of a highly evolved social system or biological organism. This parallel is useful to my purposes here because it highlights that the absence of an explicit architectural act may nonetheless result in something we all identify as an architecture.

Early London, as an example, had been the beneficiary of no grand plan, and yet a workable infrastructure was there for all to see: goods arrived, people were induced to perform needed work, wastes were carried away, water flowed to where it was needed. The varied interests of many participants combined to create a force of evolution that provides constantly improving infrastructure. No single mind ever conceived to that result, yet it happened. This is similar to what Adam Smith identified as the basic mechanism of the market: many 'invisible hands' acting together to lead to a useful outcome.

On a recent trip to India, I encountered another example of an architecture without an architect. In the city of Bombay and surroundings, an unusual set of cultural circumstances have inhibited the creation of sufficient small restaurants to serve the lunch-time needs of the working population. The result is that, since Bombay's great explosion of wealth and prosperity, the city's workers have continued to provide their own lunches from home. Lunch is a hot meal in India, and so a network of delivery men has sprung up to transport hot meals from the worker's home, (where they are prepared by family members,) to the worker's office. These delivery men are called in the local vernacular, "box-wallahs."

The box-wallah provides an insulated box to hold the meal, a labeling system to get it to its destination, a pickup and delivery service, and the same service in reverse during the afternoon to return the containers and utensils back to the home. The actual delivery involves several transfers of both boxes and funds; by the time a box arrives, it may have passed through the hands of half a dozen box-wallahs. The Bombay train stations around noon are crammed with literally thousands of box-wallahs carrying racks of boxes. To make matters a bit more complex, the box-wallahs are in general illiterate. That means the labeling scheme has to be understood by people who can't read. By the way, the same code serves in both directions: it gets the box to the office in the AM and then back to the home in the PM. The Bombay box-wallahs deliver more than 500,000 meals a day.

Now imagine you had been the architect of Bombay's box-wallah system. I list just a few of your considerations in laying out the scheme:

what is the labeling scheme?
what is the equitable payment scheme that assures all of a given box's handlers get adequately remunerated?
what happens when one of the box-wallahs doesn't show up on a given day?
how does change of location get handled?
how shall routes be allocated?
under what circumstances are routes changed?
how does each transfer work?

As you can see, this is anything but a trivial problem.

It's amazing that the system can be made to work at all, much less as well as it does. What's more amazing still is that there is no company that runs the box-wallah network, no ownership, no management and no designer. No single mind ever thought out the answers to any of the questions I listed above, nor to any of the other details that are necessary to make the network function. The entire system simply evolved.

Evolved or "organic" architecture compared to explicit architecture
Just because there is no architect and no explicit architectural act does not assure that there will be no architecture. There is always an architecture, a good one or a bad one, but there always is one. Over time, the architecture of any system tends to improve. This is a bit of a surprise, since we also know that the micro design of a system tends to worsen over time as many individual acts of maintenance make successive change more and more difficult.

The human body has evolved to have an admirable architecture. Similarly, an evolving software system can begin to exhibit some semblance of architecture. Take for example the operating system called DOS, one that is famous for its near total lack of original architecture. At the beginning, it was a strapped together set of concepts and code fragments from past operating systems, a bit of CPM, a bit of old Digital Equipment OS technology, some tape and glue. It resembled an airplane made by strapping wings onto a jet engine and tying a wooden chair on top for the pilot to sit in.

Over time, though, things changed. The very first TSR program was a complete kludge, just another element strapped on. But by the time the second and third TSR had come along, and the 200th and 300th, DOS had begun to evolve to accommodate the arrival of the 301st and subsequent TSRs. Similarly the scheme of I/O channels, largely neglected in the original PC design step (if there was one), has been rectified over time. Today, the PC can accommodate serial and parallel I/O, SCSI, and various networking interfaces. The invisible hand of the marketplace has supplied all the architecture that is in the modern descendent of the original IBM PC.

While this organic architecture does happen on its own, it is hardly what we should be trying to achieve. It takes forever and results in numerous and expensive dead-ends along the way. Worse, it may end up serving its own needs rather than ours. The opposite of an evolved architecture is an explicit original act of architecture that provides successful initial direction to a system or family of systems. The best example of this that I know is the system family that began at Xerox and ended up at Apple Computer, the family that I shall call "Dolphin-Diablo-Macintosh."
A triumph of architecture

The Dolphin-Diablo-Macintosh family has been a triumph of purposeful architecture. The family has had, from its beginning, a huge capacity to respond to changing needs. Its ability to accommodate change has surprised even those who were responsible for its architecture. Take as one tiny example the multiple screen facility that was implemented on the Macintosh in the early eighties. No one had explicitly foreseen the possibility that a Mac might someday have more than one display, but when the need was identified, the implementation proceeded rapidly and without pain. The nearly immediate result was a "display space," sometimes served by a single screen and other times by an array of up to nine. Though the screens may be of different dimensions, different aspect ratios, different resolutions, and different color depths, the scheme works flawlessly. Today I can drag a desktop icon across my whole display space, from my RGB monitor, through a multisynch monitor to my television screen, and drop it there without ever worrying that it may get lost in a crack between the screens, or that any one of my screens won't know how to display the image.

It is a trivial matter to make a design that can accommodate some change that was explicitly expected by the designer. But how do we design to accommodate unexpected change? This is the central technological challenge of system architecture. While it is a most important architectural concern, I shall offer no specific insight on the subject beyond a pointer to one marvelous book: Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides. [Reading, MA: Addison-Wesley, 1995]. In this landmark work, the authors introduce the idea of a pattern or reusable abstraction. For an investment of two hours in reading Gamma et al., I predict you will come away persuaded as I was that patterns are the key to successful architecture and an answer to the question about how to design to accommodate unexpected change.

I make this claim in order to assert (at least implicitly) that the how-to's of system architecture are not beyond us. That is an essential point for this discussion, because as long as we remain convinced that some new breakthrough is required before we can begin a regular practice of architecting new systems, we will never get on to the real problem. Before proceeding to what I believe to be the real problem, I shall take just a few lines for some definitions:

What is this thing called architecture?

An architecture is a framework for the disciplined introduction of change. This is also a pretty good definition of design. The difference between the two is that design, as we commonly use the word, applies to a single product, while architecture applies to a family of products.

Design is always concerned with capacity to accommodate change. This is true even of the extreme example of a system which is to be run once and then thrown away. It still makes sense to design such a system, because it still has to be able to accommodate one change, specifically the change that is the implementation process. (Here I portray implementation as a change from concept to reality.) A well designed system accommodates that change easily and a bad one doesn't. Just as a test of our definition, consider an even more extreme example: a system that is never to be run and need not even be built. I assert that there is no such thing as a "good" design for such a system. The concept of design makes no sense except in the context of change.

The absence of a thoughtful architectural act assures that there is no initial accommodation to the changes that will propel the product from version to version, or the change that allows the essence of one product to be carried over into other members of a product family. As Gamma, Helm, Johnson and Vlissides point out, our problems in achieving meaningful reuse are tied to this absent architectural act.

If architecture is so important and, if as Design Patterns points out, there is a workable architectural methodology, then why on earth don't we do it?

The real problem

The real problem is that architecture costs money. Before you tell me that you already knew that, that of course it costs money, that "there is no such thing as a free lunch," let me give you pause by indicating how much architecture costs. The Dolphin-Diablo-Macintosh family probably cost Xerox PARC some $30,000,000 in its initial investment. Part of this it recouped when Apple purchased some of the rights to the desktop concept in the late seventies. Considering that transfer payment and the work of its own architecture group, Apple had invested about $25 million in architecture before beginning the Lisa project. Today, if all architectural investment were capitalized, Apple would have it on the books as more than $125 million. How does that compare to any single development project at Apple that makes use of the architecture? Consider the Power PC implementation which occupied some 300 people for most of two years. The project cost around $60 million, supported by a capital investment of $125 million. The sobering fact is that a decent architecture costs as much as two full implementations. Your very first implementation will cost you about three times more with an architecture, compared to throwing the product together without paying for a single architectural thought. What we're talking here is real money.

In my consulting practice I see one organization after another engender an architecture group because they realize that architecture is the key to product family reuse, and thus a desirable goal. But then the architecture group is funded only as a part of the first implementation project, and that project's budget is set in the usual fashion for our industry, i.e., in the mid-range between Impossible and Highly Unlikely. In other words, the architecture group is a sham. It has a charter, but no funding. After a momentary digression on reuse, it is treated as an adjunct to the project and driven by the same dynamic as all the rest of the project, getting the product out the door as quickly and cheaply as possible. It is no surprise that no useful architecture ever comes out of such a group. Organizations that pay the cost of architecture get what they pay for and organizations that don't don't get anything at all. My experience is that the enormous majority of architectural efforts in our industry are in this latter category.

While I'm at it, there are two further inhibitors to architectural success, and these two are also strictly non-technical:

Early Overstaffing: Projects that have too many people on board when the critical architectural work needs to be done are forced to hurry on to the kinds of tasks (coding and testing) that can use up all those people. Early overstaffing is a sign of an organization's unwillingness to invest time in architecture.
Production Mentality: The more an organization tries to force software development into a production niche, the more wary it is of a research-like endeavor such as system architecture. Thus one well-known networking company runs a separate project for each platform, rather than develop a single cross-platform architecture. Doing the same project over and over again, building identical functionality on different platforms, is a production activity. The company can even achieve a high CMM level in doing this kind of work. Building a single cross-platform architecture, on the other hand, is a damned hard R&D project, one that scares the company silly. Such a project might even lower the organization's CMM level. So the company just can't do it. Fortunately the invisible hand of the marketplace will provide some competitor who can, and the production mentality company will go out of business (probably with CMM level 3).

A grim history lesson

The best architectures of past have all been for naught. PARC failed to recoup its full investment. Xerox's "office of the future" never made a nickel for the company. Apple has not prospered. IBM's 360 hardware architecture -a marvel of the time- and its ambitious System Network Architecture (SNA) did not herald the beginning of a new prosperity, but the end of an old one. It seems that the companies that invested in meaningful software architecture have all declined. Why is this?

Before concluding that architectural projects are the cause of this decline, I urge you to look a bit further. Architecture is a big investment. Companies that make big investments are usually successful companies, acting sensibly to become more successful in the future. Xerox's decline was not the result of its architecture, but of the success that paid for that architecture. Specifically, that success sparked a ruinous assault by the Justice Department against Xerox and eventually resulted in a consent decree that forced the company to give away its principal patents, X-1 through X-9. The main beneficiaries of this patented technology were Cannon, Sharp, Minolta and a host of small Asian printer and copier companies.

Similarly, the success that enabled IBM to develop the 360 and SNA, also set off anti-trust action against it. The result was the company was forced to release its products for "cloning," something that was unheard of only a few years before. Cloning is stealing someone else's intellectual property, something that is specifically illegal in most of the world. But IBM had to permit it, first with the 360 and then with its PC line. The beneficiaries were the Asian cloners.

Apple was also a victim of the anti-trust action against IBM. In a head to head battle against IBM, Apple might have fared well, but it could not compete against the cloners who had been given IBM's intellectual property for free.

Today, one of the most auspicious architectural endeavors is that undertaken by Microsoft. It has already produced Ole, the Foundation Classes and a host of other architectural wonders. And sure enough, the Justice Department is right there again. Who would be the beneficiaries, I wonder, if Microsoft were broken up, forced to release its investment to the public sector for the free use of all? I suspect it would again be foreign competitors.

While those who have invested most prominently in software architecture have often not reaped the expected benefit, those who have failed to invest where such investment was required have fared even worse. A striking example of this is the FAA's attempt to re-implement our national air traffic control system, the NASPLAN projects. A total failure of architectural thinking has assured that these projects had almost zero chance of success. Indeed, by any way of reckoning, the entire effort has been a fiasco. There is no redesigned Air Traffic Control System on America's horizon, and most people who have been involved in the effort in any way have acquired a psychopathic fear of flying.

Reprise

System architecture is expensive, but probably not as expensive as its absence. Today we have the capacity to build successful architectures, but often not the will. Our largest and most successful software companies have been the rare exceptions, and these have maddeningly been tageted by Justice Department actions that have resulted in loss of payback on their investments. The problems that have most often hampered architectural efforts have not been technical, but political, economic and sociological.