I received some great comments on Why Puzzles and Riddles Discriminate. Adam has a terrific list of the things he's looking for when he uses “puzzles and/or brainteasers and/or random programs to test”:
- Do they give up right off the bat?
- Do they ask questions or sit silent pondering?
- Do they make different attempts or approaches?
- What areas (if any) do they get stuck on?
- Can they explain what they are thinking? If yes – great. If no – what's the reason?
And, Craig and Eric disagree with my original post. I may not be able to convince Craig and Eric, but let me use Adam's list and explain how the answers might be different when the interviewer uses a puzzle/riddle vs. using an audition.
In the interview, the context is key to seeing how a person succeeds at work. If the context of the interview is congruent with the workplace, the interview can be a reasonable representative of how the person will interact with people at work. But if the context of the interview is different from the work, the kinds of questions and auditions are less likely to be useful.
I was working with a hiring manager and the interviewing team recently who liked giving what they called “technical puzzles” to candidates. But they'd had several problems recently: two recent hires quit, and one candidate walked out in the middle of the interview. I asked what they were doing for these technical puzzles, and was told they had several–and all of them were about developing a recursive algorithm. The problem was these folks didn't do any recursion at all. In fact, they had a huge data transformation application, so the context of the interview didn't match the context of the job. Once new hires realized they'd made a mistake, they reactivated their job search and left. And one candidate didn't bother waiting until the end of the interview. He'd flipped the bozo bit on the interviewing team.
At another client, the interviewing team liked word riddles. The team traded off who got to ask the puzzle/riddle question during the interview. (That team had 5 people, each of whom had a different puzzle/riddle.) Because the interviewing team were not great at asking behavior-description questions or using real auditions, they missed bunch of people who were hired by other managers in the company (the new hires were successful).
I was able to talk to the new hires and ask them questions to understand what the interviewing team was missing. Several of the new hires misunderstood the logic puzzle because English wasn't their first language, and they didn't hear the puzzle correctly. Two of the new hires decided that if that's how the team evaluated potential candidates, they didn't want to work with that team. And one of the new hires had attempted to explain why there was more than one solution to the riddle, but the interviewer couldn't hear that.
You might say that this interviewing team was able to hire into their culture, and you'd be partly right 🙂 But in contrast to their interview questions, this team actually had a culture of looking into a problem several ways, debating how to do things, prototyping to see what solutions could look like–a much more collaborative way of working than their puzzle/riddle questions probed. The interviewing team shortchanged themselves by not asking questions/doing auditions that reflected the way they actually worked.
You've probably noted I haven't addressed the discrimination issue here. No one would talk to me about that, because they were afraid (both the new employees and the interviewing teams) that I would be forced to go to a lawyer if I had data that said they were discriminating. But I did notice that in my clients' large North American cities, those clients who rely heavily on puzzles and riddles have primarily-white, primarily male staffs (this is in contrast to other organizations who have much more diversity). I understand about the problem attracting women to the field in general (see MusingsonWomenandIt. But I could not understand the lack of Asians and other people who aren't white.
Our choices of questions that are not directly related to the job (and no matter how you slice it, puzzles and riddles are only indirectly related to the job) reflect our culture. It's quite clear that the puzzles and riddles interviewers choose reflect their individual culture. And without meaning to, that culture primarily selects for people just like themselves.
I do want to ask questions or observe behaviors similar to Aaron's questions. But I want to see them in the context of what people do at work, specifically work in this organization. So I want to make all my questions and auditions as context-sensitive as I can. I want to know that candidates will succeed here, in this context.