My analogy is to data structures. Imagine that we didn’t have names for data structures, and you had to repeatedly say, “Let’s put data in objects where each object has a reference to the next object, and the last reference is null, and we keep track of only the first one.” It’d be hard to communicate. There’s a bunch of information in your head associated with the name linked list and without the term you wouldn’t be able to organize it or communicate effectively about it.

Design patterns are just that: names for common ways of organizing code, so instead of saying, “Let’s make a class that you can configure and then ask it to atomically make another object so that the other object doesn’t need a long constructor or ever be in a half-constructed state,” you can just say, “Let’s use the Builder Pattern.” Along with the name comes a shared understanding of how and when to implement it, advantages and drawbacks, and what to call the class.

People often crack open a pattern book and say, “There’s nothing new here!” That’s the point. It’s for code structures that we’ve been using for a long time but never had a name for. In fact, seen that way, it’s a bit ridiculous to bash design patterns. Why wouldn’t you want to name to a commonly-used thing?

In everyday conversations I try to ask the unambiguous form of the question, but when interviewing job candidates I’ve found it useful to keep it ambiguous to see which one they’ll pick. It reflects something about the way they think.

For example, I’ll ask a senior database administrator (DBA) candidate what style he uses to name tables (capitalization, prefixes, underscores, pluralization). I have a preferred answer to this, but whatever answer they give I’ll ask, “Why?” Most candidates will interpret this to mean, “What’s the cause?” and will answer “Because that’s what the style guidelines say at my company.”

At this point the interview is pretty much over, but I’ll usually ask something like, “Well what if you worked here?” The answer is usually, “I’d follow your guidelines.”

“What if you had to write the guidelines?”

“I’d use the same ones.”

“Why?” (again)

“Cause that’s what’s in the guidelines of the company I work for now.”

At no point is the candidate considering the “purpose” interpretation, which is the interpretation I’d expect from a senior candidate. He should be thinking about the benefits of the style he’s using over the alternatives. Not doing so shows a lack of introspection, and someone who doesn’t introspect won’t improve and isn’t senior.

My ideal conversation would go something like this:

“What style do you use for table names?”

“All lower case, with underscores between words, singular.”

“Why?”

“Uppercase can cause problems in some dialects because of the case-insensitive nature of the parsers, and you’ll get different results on Windows and Unix with MySQL because they use the table name for filenames. Underscores make it more readable, and singular reads better in most places (though not in the actual join clause).”

At that point the interview is also over, but in a good way.

    while ((message = map.get(key)) == null
        && System.currentTimeMillis() < timeoutTime) {

        wait(1000);
    }

The wait() call blocks the thread, waiting for the notifyAll() from the thread that puts the message into the map. The 1000 means one second. The timeout was going to be on the order of five seconds.

The above code is simple and correct. It’ll just keep looping until the value comes in or until it has timed out. The timeout may go over by up to one second, but that’s not a problem in this case. (Or rather, by the time that happens, you’ve got more serious problems.)

The code was reviewed by two other people. Both complained that the wait() should wait for the time between now and the timeout, not just one second. They argued that my code wakes up the thread five times unnecessarily. I replied that the key was very likely to be there within the first second, and that waking up a thread is not very expensive. I argued that their proposed code was more complex, and therefore more likely to have bugs.

They both said, “One subtraction is not complex!”, went back to their desks, and emailed me their modified versions, just to prove how simple it could be. Both had introduced a bug. One person’s bug was straightforward: he used the wrong constant for the calculation. But the second person’s bug was subtle:

    while ((message = map.get(key)) == null
        && System.currentTimeMillis() < timeoutTime) {

        wait(timeoutTime - System.currentTimeMillis());
    }

There’s a small chance that the current time will have advanced by the time the subtraction is done, resulting in a negative value passed to wait() and an IllegalArgumentException getting thrown. In order to save the computer one rare thread switch, he introduced a bug that would have occasionally and mysteriously caused operations to fail.

Every line of code you write is a potential bug. Do not write any line of code unless you absolutely need it right now and your program will suffer for the lack of it. Do not write routines speculatively. Do not write abstraction layers you don’t need right now. If an optimization will add any complexity whatsoever, even a subtraction, resist it. You will be sorry in five years when your code is riddled with potentially-buggy code that you never really needed to write.

The first is that many lay-offs are actually firings. Companies don’t dare fire too many people when times are good because the public may perceive the company as having problems. But in a recession the company can get rid of its 10% lowest-performing employees and blame it on the economy. Some companies do get rid of entire projects, where both good and poor employees are lost, but hiring is so difficult that a company will typically try to keep the best employees before flushing the rest.

The second reason is that good employees, those not getting laid off, hold on more tightly to their jobs. An employee might normally be thinking, “This job isn’t so great, I bet I could do better.” In a recession, though, he’ll be thankful just to be employed. His current situation is likely to seem more stable and safe than a position in another company.

During a recession, then, companies with good hiring standards must find it harder to hire, because although they are flooded with resumes, those resumes are of lower average quality.

One exception is when you max out your vacation days. Then you may as well take them or they’re lost anyway.

I unfortunately discovered this ten years ago, and ever since then my vacations have cost a lot more because I count the reduced check that I’ll get when I eventually quit.

My impatience leads me to a breadth-first programming style. I quickly write a minimal program that accomplishes the task (a quick pass of the breadth at a high level), then fill out more and more detail (descend into the depth of the tree), adding minor functionality, error checking, edge case handling, comments, documentation, refactoring, etc.

Another approach is depth-first programming, where you write your first method perfectly, handling every error condition and special case, writing copious comments, and so on. Then you move on to the second method. It’s exhausting to work this way. There’s no feedback to keep you motivated. You can’t run the program at all until you’ve written every line.

In theory both methods lead to the same program, just in a different order. In practice, though, they each have different problems. The major problem with the breadth-first approach is that, at some point, the program will be “good enough” and you’ll move on to another program. From your boss’s point of view this makes sense: there’s more value in your starting something new than polishing your current program. Programs written this way don’t bottom out the tree.

There are two problems with the depth-first approach. The first is that if you run out of time, you have nothing. This happened to a co-worker who worked for three months on a program and when the deadline came he hadn’t yet implemented the “save” feature. The rest of the program was gorgeous, but the client couldn’t use it for his project and another program was used instead. During the same three months I wrote a program breadth-first; it was used immediately and is still being used 14 years later.

The second problem is that if you realize part-way through that you designed your program poorly, or to changing requirements, then you’ve wasted a lot of time polishing routines that get thrown out altogether.

So what’s the best way to write a program? I’ve spent the last 15 years trying to move from breadth-first to depth-first, despite its two drawbacks. I find it easier to bottom-out each routine one at a time, rather than later finding the permission or self-motivation to go back and do a polish pass. The quality of the resulting program is therefore higher. The code looks better. There are fewer bugs, less unfinished business.

What about the two drawbacks? It’s true that you might run out of time, but I’ve only seen that once and presumably it’s manageable with either a flexible deadline or a conservative schedule. The second problem is wasted effort, but that’s acceptable to me if the result is of higher quality.

Avoiding “unfinished business” is important, at least in a work environment. After you’ve been at a place for a few years you are haunted by every little missing comment, missing error check, missing little feature, unpolished routine, every unhandled edge case, sloppy parsing, outdated documentation, unreproducible crash. You do nothing but apologize for and fix your old code. It’s dreadful. You leave the company in frustration and your successor convinces your manager to rewrite everything you’ve done, because “it would be easier than fixing it.”

My co-worker had the last laugh, because four years later he was writing new code while I was cursing that original program I’d written sloppily in a breadth-first way.

(I learned later that he thought I meant a truly random graph, where each day is a random number. I don’t know whether he would have taken the challenge had he known that I was referring to a random walk, which is a value that goes up or down by some random amount each day.)

He said that a six-month interval would be easiest to distinguish, so I went for 130 days, which is the number of weekdays in six months. I generated six graphs of random walks with 130 samples. I also chose six stocks off the top of my head, downloaded their history from Yahoo Finance, and for each one picked a random six-month interval. I then normalized all twelve graphs vertically.

The next day I gave him the twelve printouts and asked him to separate them into two piles: stocks and random. Other people got interested and gave it a shot. One person deliberately separated them randomly, without looking at the charts.

If done randomly, I would expect six of the twelve charts to be in the correct pile. In fact there’s a 43% chance of getting six right, out of twelve. There’s a 24% chance of getting four or eight right, 4% chance of getting two or ten, and a 0.1% chance of getting none or all twelve correct. (You’re always getting an even number right because every stock you think is random results in a random you think is a stock.)

We decided ahead of time that if he got ten or twelve right, he would be able to claim that stocks look different than a random walk. If he got four, six, or eight right, then either they look the same or he can’t tell the difference. If he got none or two, then they’re different but he’s got it backwards.

Here are the results. The day-trader is person V, who spent about ten minutes sorting the twelve charts. Persons W, X, and Y looked briefly at the charts. Person Z picked randomly without looking at the charts. The green and red colors represent correct and incorrect guesses, respectively.

The person who picked randomly got half of them right, unsurprisingly. So did another person who looked at the charts briefly. The day-trader did slightly worse than average, but still within the reasonable bounds of randomness. Surprisingly, though, two people did so badly that they imply that there is indeed a difference between stocks and a random walk. There was only a 4% chance of them getting so few right.

The right-most column shows the number of correct guesses for a given graph. Graphs E, K, and L managed to fool all four people who looked at them. (Person Z wasn’t looking.) The graphs are shown at the bottom of this page.

So what did we learn? I’m not sure we learned much. One could argue that I removed too much information from the stock data. (I graphed only opening prices, not close, low, or high.) I didn’t include any information about events, such as quarterly reports.

What I was really trying to show was that it’s not possible to reliably make money from day trading because it’s akin to trying to predict a random walk. I’m not sure that I’ve proven that. Just because it’s indistinguishable from a random walk after the fact doesn’t mean that the path of the walk couldn’t have been predicted.

I could repeat the experiment with two changes: adding candlestick marks for low, high, open, and close; and not normalizing vertically. That’s more like what a day trader would see.

But perhaps comparing to randomness is irrelevant. Maybe I should pick a real stock’s historical period, hide the last 10%, and ask people to predict the last 10% based on the first 90%. I may have to add various information about price/equity ratio and whatnot. It’s getting complicated. Or maybe I just should read Fooled by Randomness, which will just point me to the studies that have done all this already.

Below are the graphs along with some information about them. You can also download the Python script and stock data that were used to make the graphs: make_graphs.py, AIG.csv, CIT.csv, IBM.csv, MMM.csv, SWY.csv, and YHOO.csv.

A: 130-point random walk. Two out of four people thought this was a stock.

B: 130 days of SWY starting May 22, 1991. Three out of four people thought this was a random walk.

C: 130 days of YHOO starting September 7, 2005. Two out of four people thought this was a random walk.

D: 130 days of IBM starting September 10, 1980. Three out of four people thought this was a random walk.

E: 130-point random walk. All four people thought this was a stock.

F: 130 days of AIG starting October 18, 2002. Two out of four people thought this was a random walk.

G: 130-point random walk. Three out of four people thought this was a stock.

H: 130 days of CIT starting June 19, 2008. Three out of four people thought this was a random walk.

I: 130-point random walk. One out of four people thought this was a stock.

J: 130-point random walk. Three out of four people thought this was a stock.

K: 130-point random walk. All four people thought this was a stock.

L: 130 days of MMM starting December 13, 2002. All four people thought this was a random walk.

Our son Milo is starting to write, and I’m learning that lesson again. When I ask him to spell a word, such as “hat”, he’ll just come up with “H” and “T”, as if he’s spelling the whispered version of it. He’s focusing on the parts that matter.

Making this harder is that, in English, every vowel can sound like just about any other vowel. For example, the bold vowels here all sound the same: terse, first, worse, and purse. For A and O you’ve got ball and ostrich as well as pal and vowels. When Milo chooses the wrong vowel (which is about half the time), the vowel he guesses does make that sound in another word.

It won’t be long before we switch to memorization (sometimes called “sight words”). Logic can’t take you far with English.

Common sense tells us that criticizing someone will motivate them to try harder, and praising them will let them slack off. Here the praise motivated me. Why? I felt that I had built a good reputation, and keeping a good reputation is easier than building it. I was therefore motivated to never disappoint my manager. I knew that a single missed estimate would require many more good ones to regain his trust.

I use this technique on my son all the time. I’ll say things like, “You’re really good at sharing, Milo,” after he shared a toy. Even if he’s not all that good at sharing, this will (I hope) motivate him to maintain that good reputation.

I hate to floss. I suppose we all do, but I have found it particularly hard to be motivated. This is because, without exception, at every cleaning the dental hygienist tells me that I should floss more. There’s never any praise. This is even after the periods when I did a pretty good job flossing. I think if she said, “You’re good at flossing, I can see that,” I would floss a lot more. It wouldn’t even need to be true, though it shouldn’t be completely false or I’ll see through it.

Twice at dental cleanings she told me that I’m good at keeping the far ends of my teeth clean (the sides of the very back teeth). Guess which part I’m most excited about brushing?

It’s counter-intuitive, but someone who’s doing a 75% job at something can probably be motivated to improve by pretending that you think he’s doing even better than that. It’s difficult, as a manager, to do this. You feel like a 75% achiever doesn’t deserve the praise, and they don’t. But it may be the best way to get the results you want.