Why Software is Late

May 11, 2003

The Programmer’s Credo: We do these things not because they are easy, but because we thought they were going to be easy.Pinboard

I’ve seen very few large software projects finish according to their original schedule. People have suggested all sorts of reasons to explain why that is. I’d like to suggest another.

Let’s suppose that there are three types of people: optimists, realists, and pessimists. We don’t care about pessimists. I’ve always prided myself on being a realist: when evaluating a potential project, for example, I always try to think through all the little details and try to factor in a certain number of unknown problems. My estimates have usually been fairly accurate; or, at least, they’ve been too long as often as too short. I’ve also been annoyed at optimists, who always think that not only everything is possible, but everything is easy.

My experience at the company where I work, a startup, has made me rethink this. I’m mostly surrounded by optimists and I find that I spend much of my time in meetings trying to convince people that the project or feature in question will take longer and be harder than they think. I usually fail, and projects are late. But is that a bad thing?

In retrospect, if we had had any understanding of how complicated the project was going to be, we never would have started. —Daniel Hillis, founder of Thinking Machines Corporation1

Say that a large project will take two years to complete, and an optimist estimates that it will take six months. The company launches into it, and the schedule slowly slips until the project takes two years. The software is very late, but at least it’s written.

I thought it would take 1–2 months. I underestimated. It took a year. Had I known it would’ve taken that long, I might not have begun. —Philip Kaplan, founder of DistroKid2

If instead a realist had been asked, he would have estimated two years. The company may have decided against doing the project at all, especially when at the same time an optimist was being asked about a competing project and his sounded far more worth the smaller investment.

If I had any idea how hard it was going to be, I wouldn’t have done it. Ignorance is very powerful. —Jeff Bonwick, CTO of Storage for Sun3

So that’s my theory. Software projects are late because the only people who start them are optimists, who by definition underestimate how long they will take. Software projects are late by construction, not because of any failure of estimation or development. It’s a modified version of the anthropic principle.

Bitcoin looked like a hard problem, and turned out to be a harder problem than it appeared to be. Had the early contributors known how much effort it would take, they probably wouldn’t have attempted it.4

The only projects I’ve seen finish on time are the ones that were estimated by realists and were not optional. They had to be done, so it was possible to make an accurate (and discouraging) estimate and still carry on with the project.

What does that mean in practical terms? It means that it’s okay (and even necessary) if optimists start companies, launch projects, and wildly underestimate schedules. It’s okay for software to slip, because it’s not really slipping. Optimists are as valuable as realists: both are necessary in a team.

It also means that a realist will, by himself, probably not start companies or large projects. If starting a company is important to him, he must temporarily become an optimist during the idea-exploration phase. He must believe that the idea in question, though it would seem overly complicated and not particularly worthwhile to his realistic self, is actually a useful and easy project to his optimistic alter ego. It is the only way that he’ll get past the starting line and commit. The idea will later morph into something useful (initial ideas always do) and the schedule will slip, and that’s okay.

Update 1/10/2006: A co-worker (one of the optimists) read this essay and pointed out that realists aren’t actually necessary in a team. But publishing software isn’t done in isolation. You need to synchronize with marketing, sales, PR, and in general have your product launch all ready to go when the software is done. A realist can help make an accurate software schedule so that the big marketing splash doesn’t happen two months before the software is ready.

1“Richard Feynman and The Connection Machine”

2“What I’ve been up to for the past 4 years”

3“Free Your Mind: How a Small, Persistent Team Created a Revolutionary File System” (no longer online)

4“The Truth About Theranos”