Pernicious Complexity

Bruce Schneier linked to Marcus Ranum's "The Six Dumbest Ideas in Computer Security". It's all interesting, but what sets me off this morning is this sentence from the discussion on Schneier's page, in which Ranum says

We can make systems that are more powerful but less complex. I absolutely believe that to be true. It is, however, easier to build systems that are more complex and more powerful.

That's the problem.

Joseph Tainter's 1996 paper, "Complexity, Problem Solving, and Sustainable Societies", beautifully lays out the consequences of the lure of utility at the expense of complexity. [link] [Wikipedia] [Google]

Tainter's paper does not even contain the word "computer" (it's about energy and sustainability) but the central theme applies perfectly to the computer world. The point of Tainter's paper is that complexity and utility are related, and that the relationship over time is predictable. It starts out where you fix a problem by adding some complexity and get a great deal of utility in the process. Later on, when you have another problem you've got also to deal with the complexity you added before. Things are a little harder now, but you add more complexity because you can get more utility. Things continue this way for a while.

Eventually though (and this is where Ranum trails off) you start getting less and less utility for each increment of complexity, until it reaches the point where you're just dealing with the complexity and keeping things afloat somehow. The utility/complexity curve has levelled off. If you stay on this track, if you keep doing what you're doing, you'll start heading down the back side of the curve, adding more complexity to keep things going, but losing utility. Now you're on the road to eventual failure.

Read the paper. I'm not doing it justice.

If I were in charge of something like an IT shop, nobody would work for me that had not internalized Tainter's message. If I were in charge, internal customer service would "suffer" because internal customers would no longer automatically get what they want. All requests for new functionality, and all proposed approaches to existing problems, would have to be examined with respect to their impact on complexity. Very little would be adopted unless it somehow resulted in a reduction in effective complexity.

Of course, I wouldn't be in charge for long.

Read Tainter's paper.