“I took a slightly different approach when I was a hiring manager. Instead of canned set of questions, I would get the candidate to a whiteboard, pose a vaguely-stated, open ended design problem with no “right” answer, then ask the candidate to walk me through how they would approach it. Depending on what skills I was looking for, I'd ask for Use Cases, specific UML diagrams, etc. Since the problem was vague, I expected that candidates would ask for clarification (rather than diving right in), and by tossing in new information midway, I could get a handle on whether the candidate could switch gears, or whether they would charge ahead with whatever approach they'd started off with.”
“Looking at what level the candidate chose for their initial approach was also enlightening. Architect/Designer candidates who started sketching data structures before they'd clarified the problem flunked out rather quickly. (Even when hiring developers, this approach was good for getting a quick grip on their true level.)
“I wrote the technique up a while back. See http://www.perlmonks.org/index.pl?node_id=53569.” [JR Note: Dave's post is slightly perl-centric, but the ideas are clear.]
Dave shows a very high level of interviewing competence here. He starts with an open audition, and then tailors the audition on-the-fly to progressively probe the candidate's abilities. Dave looked for information early: “candidates who started sketching data structures before they'd clarified the problem flunked out rather quickly,” and Dave was able progressively filter out candidates based on their responses to each part of the audition.
If you're a hiring manager, think about how you can frame auditions so that you can detect the candidate's level of job ability. If you're a candidate, think about how you want to showcase your abilities to hiring managers.
If you're a non–technical manager of technical people, work with your technical interviewers to create several auditions like Dave's, and then watch them. Part of the audition might be for the candidate to explain to you what they're considering, especially if you're going to ask employees to explain what they're thinking to you.
You've probably noticed that auditions play a huge role in successful knowledge-worker interviews. If you're not comfortable with auditions, learn how to create them and use them. Your hiring will be much more productive and effective.