The Signs of Trouble: On Patterns, Humbleness and Lisp

by Adam Tornhill, October 2012

Our challenge of humbleness

In his classic talk at the Turing awards, Dijkstra remarked that computer programming is an "intellectual challenge which will be without precedent in the cultural history of mankind" [1]. What is it that makes software development so hard? Dijkstra himself gave the answer by concluding that a "competent programmer is fully aware of the strictly limited size of his own skull" [1]. It's a reference made to the great cognitive challenges of software development. Programming stretches our cognitive abilities to a maximum and we need to counter with effective design strategies to handle the complexity inherent in software.

One such strategy is to share knowledge and base our solutions on what has been known to work well in the past. Since few designs are really novel we often find that previous solutions, at least on a conceptual level, apply to our new problem too. Another strategy recognizes the capacity limits of our brain. Identifying ways to break those limits provide for more efficient usage of our precious grayware.

Patterns incorporate both of these strategies. As software developer and author of a technical book on patterns I obviously find value in the pattern format. And as a psychologist I see the links to our cognitive capabilities and the social value of patterns.

In this article I will detail my view on patterns and the value I see in them. Since patterns are a controversial topic, I will build the article around the criticism against patterns. Let the critics have the first word.

Misunderstood, misused or a sign of trouble?

A common view, expressed here by Jeff Atwood of Coding Horror fame, is that "design patterns are a form of complexity" [2]. As such, patterns should be avoided when simpler solutions would do.

Another view that tends to pop-up in discussions are patterns as workarounds for missing language features. I will ignore for a moment that patterns aren't limited to the programming task itself - patterns have been harvested in fields as diverse as testing, team organization, databases, etc - and keep the discussion on the subset of patterns related to design.

Proponents of this view refer to Peter Norvig's presentation on design patterns in dynamic programming languages [5]. In his presentation, Norvig classifies 16 of the 23 design patterns in the seminal Design Patterns book [6] as simpler or even invisible in higher-level languages. It's an interesting read and a true testimony to the power of the Lisp family of languages.

Paul Graham adds an interesting twist to the subject: "When I see patterns in my programs, I consider it a sign of trouble" [3]. Graham continues to express that "a program should reflect only the problem it need to solve" [3] which brings us to the very essence of design. Since I'm a Lisp programmer myself, I understand Graham's view. I also think he misses the most valuable part of what patterns really are.

The complexity of simplicity

Let me start by addressing Atwood's claim since it's more superficial than Graham's and provides a better starting point. I will work my way back through the mists of misconceptions and valid criticisms of patterns to finally take on Graham's critique from the view of a fellow Lisp programmer. But first we need to consider a more fundamental aspect: the use and possible abuse of patterns.

Atwood has a point; patterns are indeed used in situations where they don't apply. Sometimes that's a good sign. At least in a supportive environment. A willingness to try something new is a sign of intrinsic motivation. It's an attempt to improve. We need to make those errors and learn from them. The challenge is to provide an environment where the consequences are controlled. A mix of mentoring and peer reviews has proven to be a successful approach. Direct feedback is an important learning tool.

A worse problem than a motivated individual on a learning trail is over-engineering. Patterns aren't necessarily good. Used in the wrong context where the forces are unbalanced they make the resulting context worse. Patterns, like any other design choice, imply a trade-off. We buy flexibility in one area traded for some other consequences. Unless the flexibility is required we travel into the obscurity of speculative generality. A simpler solution would probably serve both the business and the brain of the maintenance programmer better. Or, as I put it in my book: "maintaining an AbstractFactorySingletonDecoratorBuilderPrototypeFlyweight isn't why I went into programming" [4]. Believe me, I've been there. There's just no supplement to good taste.

But the story is not as simple as Jeff Atwood implies. A pattern doesn't equal some automatic increase in complexity. I tend to use patterns as targets for refactorings. The reason I refactor is to get rid of accidental complexity and adapt the design to an increased understanding of both problem and solution space.

When used as targets of refactoring, patterns do shine. One of the reasons is that the resulting context is well documented. It's possible to reason about the changes up-front and contrast them to the existing implementation. A refactoring is an investment that we want to pay-off. Any good pattern description acknowledges the weak sides of the pattern, its trade-offs and hints at scenarios where the solution doesn't apply.

Knowing the common patterns in your domain gives you a powerful cognitive tool for large-scale refactorings.

Patterns as communication tools

Patterns have social value too. The format arose to enable collaborative construction using a shared vocabulary. In Patterns in C I write on the groundbreaking work of architect Christopher Alexander:

"The patterns found in Alexander's books are simple and elegant formulations on how to solve recurring problems in different contexts. [...] His work is a praise of collaborative construction guided by a shared language - a pattern language. To Alexander, such a language is a generative, non-mechanical construct. It's a language with the power to evolve and grow. As such, patterns are more of a communication tool than technical solutions." [4]

Patterns in the original sense are context-dependent and do not by some work of magic provide universally "good" designs. You can't take the human out of the design loop. This is why I get disappointed every time I see someone offering a "patterns code library" or expensive case tools that come stacked with "UML pattern templates". Reducing patterns to mechanisms doesn't tell the full story. The most interesting part of a pattern is rarely the implementation.

I understand the marketing perspective but those ideas completely miss the very essence and purpose of patterns. Sure, in software we are free from the physics that constrained Christopher Alexander. We may take a natural departure from Alexander's idea and provide general parameterizable implementations of specific patterns. If we do our job right that abstraction may well be usable, yes, sometimes even re-usable. But the ROI is diminishing fast.

Jeff Atwood falls into this trap as he labels the whole idea "a complex recipe of design patterns" [2]. Design patterns are not a "recipe" either and were never intended as such. But Jeff's view is an understandable outcome of skimming through Design Patterns by the Gang of Four [6]. One of the main problems I have with the Gang of Four book is its prominent use of class diagrams. Open any pattern in the book. First thing we see is a class diagram.

I addressed this in Patterns in C: "This has lead many developers to confuse the diagram with the actual pattern. It's not. At best, it's one possible way to implement the pattern in a certain language. Nothing more. A pattern is a dynamic, generative entity. Depending on context, the applications of a pattern may look radically different each time. It's my firm belief that much of the harm done to the design patterns movement could have been avoided had the Gang of Four just excluded the section on structure" [4].

Patterns for cognition

Patterns are primarily about communication. But the value of a shared vocabulary goes beyond communication. Patterns are powerful reasoning tools. Instead of reasoning about individual design elements and coding constructs, patterns provide a way to group these concepts into a larger unit. This has implications on our problem solving abilities.

In the introduction I quoted Dijkstra famously referring to the limits of our skulls. One of the main factors behind the limitation is working memory. Working memory is understood as the system that allows us to hold information in our mind, integrate different parts, reason about them and manipulate them. Working memory is what we use when we try to decipher a macro in Lisp, understand the relationship between two Java objects, or find a way to express a certain domain rule in Haskell.

Working memory is vital to our reasoning, problem solving, and decision making. It's also strictly limited in its capacity. Back in 1956, George Miller made the first quantification of our working memory capacity. Miller arrived at the now well-known heuristic of seven items, plus minus two. Sub-sequent research has refined Miller's number and distinguishes between verbal and visual information. The latter is even more limited with a mere four simultaneous items.

Given the few items we can hold in working memory simultaneously, it's no wonder that programming is hard; any interesting programming problem has a multitude of fine-grained parameters and possible alternatives. One way around this limitation is a process known as chunking. Chunking is an encoding strategy where individual elements are grouped into higher-level groups, chunks. While the limit on the number of units still apply, each unit now holds more information.

Patterns are a sophisticated form of chunking. Their names serve as handles to the vast knowledge stored in our long-term memory. Reading the name of a known pattern activates the associated network of knowledge and brings the ideas to conscious attention with economic usage of our working memory.

Program close to the domain

Given all the benefits of patterns, why would Paul Graham consider them "a sign of trouble" [3]? Well, Graham doesn't actually discuss patterns. He mentions them in a more general discussion of the relative power of different programming languages: "regularity in the code is a sign, to me at least, that I'm using abstractions that aren't powerful enough" [3].

So Graham discusses patterns in the repetitive semantic sense of the word. And I completely agree. Repetitive patterns as in subtly duplicated code that cannot be refactored away is a sure sign that the language lacks expressive power. I often see that in Java (consider its dysfunctional try-catch-finally pattern in versions prior to Java 7).

Patterns as a medium for sharing knowledge, ideas and used as reasoning tools are a different story though. It has nothing to do with repetitive code. The two cases are orthogonal.

In this sense, the interesting thing is not patterns within a program; it's the conceptual patterns between programs. Such patterns are a sign of knowledge, learning and propagation of experience.

The patterns found in different languages will often differ. Design patterns are a result of the transition from problem to solution domain. This is never a perfect fit. If we can evolve and grow our language, like we do in Lisp, we get closer to the problem domain. Using Lisp as building material for domain-specific languages is a powerful strategy. It gives a much smother transition now that the solution domain has been designed to express precisely the specific problem at hand. At least on the surface. Our domain-specific language, however high-level it may be, will have to bridge the gap for us. The deeper we dive into the layers of the domain-specific language, the further away we get from the problem domain. It is here that documented patterns help. In case of Lisp, the patterns would focus on language creation, capture common approaches and document the trade-offs.

Level of scale

Patterns don't have to be large and complex. They exist at all level of scales. High-level languages grow their idioms too. One challenge of learning a new language is to get accustomed with its idiomatic ways of problem solving. By documenting the idioms we can provide guidance and ease the learning curve for newcomers to our technologies. Experts may benefit from the nuances and documented consequences present in all well-written patterns. This is what I tried to explore in Patterns in C [4].

Finally, it's worth pointing out that the GoF book [6] was a starting point, not the final word on patterns. The book is almost two decades old and in desperate need of a second edition. It's deliberately focused on a limited sub-set of the design space available to programmers (single-dispatch object-oriented design). Harvesting patterns in different paradigms would certainly result in a different catalogue with different implementation mechanisms. This journey is well worth to embark on.

Interested in patterns? Programming in C? Check out my new book on Patterns in C.

References

Dijkstra (1972). The Humble Programmer
Jeff Atwood (2007). Rethinking Design Patterns
Paul Graham (2004). Hackers and Painters
Adam Tornhill (2012). Patterns in C: Patterns, Idioms and Design Principles.
Peter Norvig (1998). Design Patterns in Dynamic Languages.
Gamma et al (1994). Design Patterns: Elements of Reusable Object-Oriented Software