I had dinner recently with someone who said, “We put our candidates through the wringer. First we test them, then we interview them, then we give them another test.” I spoke with someone else whose manager wants an extensive role play to pit candidates against each other (sounds a little like The Apprentice to me). And Keith Ray pointed me towards Permutation Algorithms and Job Interview Questions (scroll down a bit to get to this part).
In addition to the puzzle questions, all of these are techniques to trip candidates, not true auditions. OK, if you are having trouble finding people who know your specific language and you can't take the time or money to train them (why the heck not?), then maybe some sort of test is acceptable. But not a whiteboard test. People program on a computer. If you want to test their knowledge of a programming language, make the test something they do on a computer. And don't kid yourself, the test is not an audition. Solving puzzles, particpating in role plays, developing any kind of software that doesn't reflect your product — all of those are tests — are not auditions. They tell you little or nothing about how the person will work at work.I liked what Kaplan said in his essay
I don't think quickly on my feet and I get “stage fright”. Even for algorithms that I've written many times, like in-order binary tree traversal, I've frozen and been unable to recall the algorithm.
Most of us are nervous in an interview and can't always quickly recall our accomplishments. Rapid-fire interrogations, puzzles, programming algorithms you can look up — all of those are intellectually interesting and not suitable for auditions.If you want to see how a person will work at work, develop a real audition — a technique to see the person's work behaviors. If you want a programming problem solved, give the person a computer and access to whatever tools that person needs. If you want to extend a design, ok, that's something that can be done at a whiteboard. (Don't you find design generally a discussion among people? Wouldn't you participate in that too?) But whatever you do, give the candidate time to think. People don't have to solve problems on their feet at work — they have time to think. The faster you want to move, the more time you need to think. When you remove the access to generally available software libraries, time to think, or even discussions on the web, you're removing the very tools a fine candidate may use daily at work. Why would you do that?
When I test my students in my classes, I use essay questions, because I want to know what they've learned. The essay questions take longer and are harder to grade, but they give me insight into what students learned (and what I didn't teach). Behavior-description questions in an interview are similar. And, auditions, are a form of behavior-description question, except that you can watch the behavior or discuss the results of some work. Don't make your interviews like multiple-choice tests. You'll get people who can talk the talk, but may or may not be able to do the work.
The problem for me as the prospective employee, is that one of the critical factors of success is the personality mesh between me and the rest of the team. And like some of these other factors you mention, that can’t be determined in a set of interviews and tests. It either works itself out over the first couple of months or it doesn’t.
Maybe that’s part of what annoys technical managers, the process they’re overseeing is inherently fuzzy. Probably why I’ve always been better at being an engineer than a manager.