Don’t Ask About Physics

I met a software developer recently, who studied physics as an undergrad. He’s now working in an IT organization on financial processing software.

He’s part of the interviewing team for his organization. They’re trying to hire 6-7 more developers before the end of the year. He told me, “I like to ask a question about physics, to see how smart the candidates are.” I asked him how many candidates he’d rejected due to his question. “Only 2 out of 5.”

Ouch. He rejected 2 potential candidates not because of an answer that’s relevant to the job, but to an  answer that is irrelevant to the job.

Instead of asking a question that you think will get you information about how smart a candidate is, ask questions that really tell you what you need to know.

  • “Tell me about a time you had to learn an application quickly. What did you do?”
  • “Tell me about a time you had to bring someone else up to speed on a system. What did you do?”
  • “Tell me about a time you got stuck on a problem. What did you do?”

All of these questions are much better than asking a candidate about physics, art history, Spanish, or anything else you took in school. And, they’re relevant to the job.

Don’t ask about physics. Ask about job-relevant experiences. You won’t be falsely rejecting potential candidates. And you won’t be opening yourself up to a lawsuit about discrimination. Ask about issues relevant to the job you have open now, not experiences you had in school.

10 Replies to “Don’t Ask About Physics”

  1. Pingback: software » Don't Ask About Physics
  2. I’d like to play Devil’s Advocate for a minute. I think it depends a lot on the question and how you interpret the answer.

    If I ask, “In what year was the Treaty of Westphalia signed,” all it tells me is whether the person is a history buff. If I ask what the speed of light in a vacuum is, then yeah, I’m just testing trivia.

    However, I can also see a physics-type question being very strongly related to assessing someone’s logical mindset. While this is not the sole qualifier, I have never met or worked with a solid developer who did not have a very strong (i.e. far above average) logical reasoning inclination.

    The problem with questions like “tell me about a time you had to solve a hard problem” is that they’re very indirect, and can tell you a lot more about a person’s interviewing ability than anything else. This is one of the reasons why Microsoft became famous for their wild interview questions.

    I will say that, ceteris paribus, software development is one of those very few career tracks in which raw cognitive ability really is a primary predictor of performance. I’d be interested to know what the physics question was and the answers the candidates gave.

  3. Not knowing unrelated subjects or facts should not cause a rejection at an interview, but not being able to reason is a valid reason to reject a candidate.

  4. Colin,

    That is a reasonable request, to ask a candidate to demonstrate logical thinking. Why can’t the question be relevant to the work that’s being done by the company, or some related work? Aren’t there enough questions to ask in those areas that it’s not necessary to ask something that the interviewee may not have thought about in many years? The interviewee will more likely have thought about some issues pertaining to the position in question because he or she wants to get that job. Something unrelated could very well catch them off guard, which is not the same as being unable to figure out something logically but indicates time spent away from a particular problem domain.

    At least this was the case when I was younger. Software projects did not massively fail any more back then than they do now. I don’t understand why there is this mindset in the software industry that doesn’t trust its ability to assess candidates based on reasoning about the subject itself.

  5. Greg,

    A very experienced auto mechanic might know how to fix a particular problem because, having seen a dozen like it over the years, he knows three places to look right away, where an apprentice might not. However, a very talented newbie might arrive at the problem very quickly due to his or her ability to reason through it in more effective ways. Ultimately, this person could become a far more productive and valuable worker.

    There are fields in which experience can substitute effectively for a lot of raw talent. The FAA, for instance, has found that while pilots’ abilities deteriorate with age, their safety record does not, suggesting that the wisdom they bring allows them to perhaps avoid entirely situations that a less experienced pilot might have a better chance of fighting their way out of.

    Sales is another discipline which by its nature seems to reward a lot of things other than general intelligence, which I think accounts for a lot of why sales and engineering seem to be like cats and dogs in many organizations.

    I’ve had many experiences in the IT field with people who were very experienced and knowledgeable in a very specific area, who fumbled terribly if pushed outside of it. I think in particular of a past life where I did a lot of very nasty work with SGML, and we had a person who was a wizard with DSSSL stylesheets, DTDs, and a whole host of arcane but closely-related things. He could work circles around any of the engineering folks when it came to solving a specific client problem with these tools, but if you asked him to write an SQL query or a few lines of Java code, it was like waving kryptonite in front of Superman. He clearly wasn’t a dumb guy, and yet…

    I think that if you’re hiring a Linux sysadmin, or someone to run a big Oracle farm, or anything else that involves a fairly stable and extremely deep platform, experience can easily trump general intelligence, because the wise old hand knows where all the traps are. However, in systems development, where you are creating new problems rather than solving old ones, the ability to process large amounts of new information seems to take over once a person has developed a baseline of professional skill.

    Where I think a degree of chauvinism may perhaps exist is in the failure of IT leaders to differentiate between these situations and hire accordingly. Of course, experience AND talent are best, and I know and have worked with many great developers who have more than a few gray hairs, and larger-scale technical leadership absolutely relies upon and rewards experience.

    Coincidentally, what I’ve found to be one of the most effective indicators of this sort of thing is to ask what people do for fun on their own time. When great salespeople go home on Friday, they golf. When great programmers lean back and relax, surprisingly often it’s to write some code. It’s really the only field I’ve run into where there seems to be so little distinction between work and play.

  6. Colin,

    There are several ways I might respond to your question, but for now, I’ll stick to my original response. Are questions that come from outside of the scope of the job the candidate is expected to do the only way to determine the candidate’s qualifications? Are there no questions that can be asked that can be drawn from practical (but not “cookbook”) problems a software engineer might face?

    In this particular case, the candidate is applying for a position writing, or perhaps supporting, some financial application. Are the chances that this company may be developing 3-D graphics software for games, or aerodynamic simulators, so great that it’s necessary to ask physics problems instead of problems that pertain to the development of financial applications? What about the various types of design issues that come into consideration with database layout or numeric representation?

    If we must use physics, why not art history, or the other suggestions JR made? Are we so sure that these problem domains do not better differentiate candiates? I can make an argument, based on my background and experience, that I should ask questions from Euclidean geometry, or advanced calculus, to determine candidate qualifications? Am I going to learn something from these domains that will tell me how the candidate will deal with transaction rollback or database integrity?

    The software industry wants to be taken seriously, like its brethren engineering disciplines (e.g. civil, aeronautical). Yet it does everything to keep anyone from doing so, because it does not trust the very principles upon which it is built to test people on them. For example, awhile back, someone posted to a Joel on Software forum that he was thinking of asking interviewees to put together chairs from instruction manuals, because there was no other way he could determine who were the best candidates.

  7. Greg,

    As I said in the beginning, I don’t mean to defend the use of a question that requires deep or unusual physics domain knowledge. Since we don’t actually know any more than what Johanna said, for all I know I’d agree with you in this case.

    When we’re seeking to determine logical reasoning skills, there is absolutely value in using context-free questions. IQ tests are designed in this way to remove as many language, cultural, and education barriers as possible. In this regard, asking a question that involves the domain is actually introducing the possibility of bias in the responses.

    Art history would be less likely to be relevant since it generally involves judgment in areas other than pure logic and reasoning. Of course, it might be interesting to ask someone to critique a painting or sculpture for other reasons, but this would definitely coming at things very sideways.

  8. Colin,

    In this case, what class of physics questions are context-free, and do not introduce bias? I imagine what is asked will be somewhat more involved than something like “if you drop a dime and a penny at the same time, which hits the ground first.”

    Furthermore, is there a reason why there should be no bias towards the domain of software? After all, people are being interviewed to write code, not be interviewing jockeys.

    FWIW, I have no problem with pure logic questions on interviews, provided that those are actually asked, and it is very clear how to prepare for such questions. If this were the case, a market would develop for logic question interview prep, with lots of practice material available. As is the case with MCATs, LSATs, and other industry licensing exams, people learn from the sample exams, and some do well. The same would be true for computer jobs. But the software industry will not commit to any standardization; rather, it just hums and haws about software is art, etc. Furthermore, they complain they can’t find people to do the work, on the grounds that there are insufficient people with the proper math and science skills (despite contrary evidence). The great irony is that the software industry is losing people to these other professions that can give much better accounts to the US government and prospective candidates of what is expected.

  9. Good post. I’ve known some of my superstar programming friends end an interview early and leave if they were faced with a stupid puzzle question, or too many questions that were completely unrelated to the work they were dealing with. There are some generic problem solving questions they were happy to answer, but experience has shown them they were wasting their time with a lot of the popular puzzle or other irrelevant questions. My favorite irrelevant question: “What kind of kitchen utensil would you be?” Answer: “Whatever I can stab myself in the eye with to end this interview early.”

    They use these kinds of questions as fleecing mechanisms to tell them that this is not a company they want to work for. “Oh. You want to know how to move Mt. Fuji? Waste someone else’s time thanks.” One of my friends was famous for this line: “Thanks anyway for the interview.” (He would then pack up and leave.) It was the company’s loss for having such poor interviewers, or maybe it truly was an accurate indicator for cluelessness and my friends were right to react the way they did.

    Often they say that they think the interviewer is more interested in showing their knowledge in an unrelated area, or with a puzzle question than in them. “HaHa! I fooled you, silly interviewee, now let me extol the virtues of my massive intelligence (read ego) upon you.” Of course not everyone is like this, but they use these kinds of questions as warning signs – and there are very few interviewees can pick up on in an interview to decide whether this is a company you want to work for or not.

    I lightly touched on one of their experiences here:
    http://www.kohl.ca/blog/archives/000176.html
    and a trick to game the system with some of those questions.

    After the post, I got two emails from companies asking for a name so they could send a job query to the person I described. 🙂 He was already happily employed at a reasonably clue-full shop, after walking out of his puzzle question interview process in disgust.

    -Jonathan

  10. People with Physics background seem to be more skeptical about “Tell me about a time you …” type of questions ( I also have degree (graduate) in Physics and work for Financial company). One reason could be that they think: “talk is cheap and easy to fake”. There is a limited number of “Tell me about a time you …” questions. An experienced job seeker can prepare for an interview by making up stories about a few “past” work situations and using these stories to answer behavioral questions. As the result you will hire a candidate who KNOWS how to behave, but not necessarily who BEHAVES. It is important difference, is not it? Technical problems/puzzles do not have this drawback, there are so many of them and a small modification makes the knowledge of the answer useless. And they give the hiring manager an opportunity to observe REAL behavior instead of listening a Fairy Tail about “saving my previous company three hundred thousand dollars”.

    As far as physics (or Spanish language) questions are concern, I think they should not be asked during software developer interview unless physics belongs to company’s business domain. At the same time, simple logic and simple math skills are usually relevant in any business, and they do not discriminate against non-physics majors. They are also widely accepted (e.g. GRE General or GMAT exams) as a criteria of smartness, which reduces litigation opportunities for rejected candidates.

    “Whatever I can stab myself in the eye with to end this interview early.” – I love this answer, because it shows where an irrelevant question really helps. As a manager, do you really want to hire an employee who gets mad after the first question he perceives as irrelevant?

Leave a Reply