There are two ways to architect a program and write code: top-down and bottom-up.
With top-down design you start with a vision of the whole program, perhaps what some of the components are, and how they fit together, but the components themselves are still fuzzy. You implement a high-level version of the program that calls simplistic versions of the components (that might do nothing) and slowly work your way downward into the details of each component.
With bottom-up design you start with the components, which you see clearly, but it’s not yet clear how they fit together. You write the components individually, unit-test them, then assemble them into a whole program.
The correct way to architect and write a program is top-down. This is not a matter of taste or preference. Bottom-up design is fundamentally busted and you shouldn’t use it. Every system I’ve been involved in that used top-down succeeded and those that used bottom-up failed. Edsger Dijkstra wrote an entire monologue about this. I’ve never seen an architect advocate bottom-up.
When you write bottom-up code, you’re writing the components before you understand how they fit together. There are two problems with this, a small one and a large one:
The small problem is that the components might not be right for the job. The API may not be right, or perhaps the way the component works isn’t right, or maybe the component isn’t needed at all. By writing the component first, you risk wasting time.
The large problem, and this is the killer of bottom-up programming, is that you end up with programs that are too complex. As you develop a component, you’re not sure how it’ll be used, so you’re tempted to over-generalize. General solutions are more complex than specific ones. When the component is finally plugged into the whole program, and only a specific use is needed, the extra complexity is never ripped out; it stays there forever, adding code, weight, and complexity unnecessarily. And complexity is what kills large programs.
Let’s say that you’re designing a large website, an online shop for a large corporation. With top-down programming you start by (for example) importing Tomcat, writing a simple mostly-empty servlet, and having it write “Hello world!” You now have an end-to-end website. You can compile it, try it, and demo it. You can even deploy it to production. It’s buggy, since it displays “Hello world!” instead of items that customers can purchase. You slowly fix that bug by adding a template engine and a simple home page template. Then you add a database call to get a set of items, which you list in the template. Then you add a shopping cart. Then a way to sign in. At every step you have a shippable incomplete site, and at every step the system is as simple as it can be to do what it’s doing. It’s clear what to do next, and it’s clear what’s needed of the next component.
With bottom-up programming, you start thinking about the components you’ll end up with. You’ll need a way to decide which page is being displayed, a way to get data from back-end services, and a way to render templates. Let’s look at templates. Which template engine will you use? Not sure yet, so better make it pluggable with an abstraction layer. How will the template know where to write its output? You haven’t even chosen a web container yet. Better add an abstraction layer for that, and for the inputs to the template. How will you get information about the request, such as the page type, cookies, language, device type, etc.? Better create a pluggable extensible system for gathering arbitrary data and storing it in a generic context object! When you go to put all these pieces together, they each do too much, are too complex, and are too hard to understand and extend. Nearly all of the generality ends up unused, and unremoved, forever.
Smart engineers are naturally attracted to bottom-up programming. Designing a component is a small tractable task that can be finished and called done. You’re creating a perfect, beautiful, reusable jewel. All engineers really want to do is write components. The top-down approach doesn’t have this nice property; the product is forever incomplete. Yes, top-down can be less appealing to some people. Do it anyway.
Large companies are especially prone to unconsciously using a bottom-up approach. It takes a strong personality to take responsibility for the application top-down, to build enough of it to see the components, then farm out the components to different teams. It’s easier, organizationally, to guess at the components, farm them out, and hope that they’ll integrate in the end. This is why so many large projects fail, why so many of these components are trashed before they’re ever used, and why large companies can’t ever seem to make much progress.
For the individual programmer, modern tools also encourage bottom-up programming. In IDEs like Eclipse, for example, writing code is easier if the types and methods you’re calling are already defined. But if you defined those first, then you’re doing bottom-up programming! Writing top-down code in an IDE causes it to bleed red until you can get around to implementing those methods. There’s persistent pressure to write the called function first.
And finally, there’s unit testing. When writing components bottom-up, there’s no application to try them in, so you’re forced to write unit tests to make sure they do the right thing. This is good. With top-down programming, you can just run the application to see if the component works. There’s less pressure to write tests, and in fact the component may not be written in a way that makes tests easy to write. That’s an unfortunate side-effect of top-down programming, but I’m not generally bullish on unit tests anyway; most of them don’t provide any value. (See Mostly avoid unit tests.) For those tests that would provide value, write them anyway, despite the fact that the component is working for the application as it is now.
At every level there’s pressure to do bottom-up programming. Avoid it.
Instead, start at the top, with
main() or its equivalent,
and write it as if you had all the parts already written. Get that
to look right. Stub out or hard-code the parts until you can get it
to compile and run. Then slowly move your way down, keeping everything
as brutally simple as you can. Don’t write a line of code that isn’t
solving a problem you have right now. Then you may have a chance
of succeeding in writing a large, working, long-lived program.