Watching Testers In Action: Auditions in Interviews

If you’ve been interviewing for a limited number of open positions, you know how hard it is to decide between two great candidates. If you’re already asking behavior-description questions about work and cultural fit, consider adding auditions to your interviewing toolbox.

Watch a Candidate in Action
Auditions help you see a candidate work—either literally or the output he or she is capable of producing.

Define Behaviors
Identify the skills you want candidates to display to ensure a successful hire. First, make sure you define the activities and deliverables for the job. If you’ve never written a job description before, here’s a good way to start: write down the names of each person with whom the candidate will work. If you work in a larger organization, write down the group names, separating the people or groups by what the new hire will deliver.

Next, define what each person would expect of a candidate, specifically defining the deliverables for each case. Finally, define the behaviors or the vehicles that produce those deliverables from that role. Here is an example for a tester position:

Person With Whom Tester WorksActivities and Deliverables Expected of a TesterExpected Behaviors of a Tester
Test ManagerGather and report data about testing status and obstacles, and develop test plans.Write defect reports, verbally advocate for certain defects, organize and collect data, compile test logs, and plan tests for area(s).
Other testersDiscuss test strategies and review test plans.Review testing strategies and discuss test techniques.
Project managerReport test data for specific area(s).Report data and information
DevelopersWrite defect reports, review designs, and develop tests for iterations.Discuss defects, review designs, comment on findings, quickly develop small tests, and organize tests.
Release engineerCheck tests into configuration management system.Know when to branch and how to access test and product source code.

Select Behaviors for Audition(s)
Sometimes I know what behavior I want to see in action, which makes defining the audition easy—I define the audition around that behavior. But, more often than not, I’ll want to see more than one behavior. In this example, I’ll develop auditions for two behaviors: review of test strategies and ability to develop little tests quickly. I want to verify a candidate understands a variety of testing techniques and can quickly develop a variety of small tests. Using these two distinct behaviors, I’ll develop a mini-audition for the phone screen and a longer, 20-30 minute audition for the in-person interview.

Develop the Audition
For the phone screen audition, I choose the behavior of “quickly developing small tests.” Here are two options for the audition:

  1. If the candidate has web access during the phone screen, I direct him or her to the product (or another relevant product) on the web, ask the candidate to visit a specific URL, and develop a list of tests for that URL. I’ll ask the candidate to discuss the variety of tests and the choices the candidate made in selecting those tests.
  2. If I know in advance the candidate will not have web access during the phone screen, set up the same audition, and ask the candidate to spend about 5-10 minutes generating test options before our scheduled phone screen. Then I ask the candidate to explain his or her choices about tests.

If you chose not to ask the candidate to perform a phone-screen audition, you can still employ the audition during an in-person interview.

Before I audition a candidate, I first verify he or she has some knowledge of the behaviors I’m looking for. I’ll ask behavior-description questions, to verify some knowledge of test techniques, such as, “Give me an example of a time you tested a product in multiple ways.” If I hear an answer indicating the person has basic knowledge, then I know I’ll learn something from the audition.

Create an audition by supplying the tester with either a part of your product, a design, or specs—whatever testers use in your organization. Ask the candidate to identify all the techniques he or she would use to test that product.

Phrase the Audition Request Carefully
Make sure when you develop the audition that you’re not leading the candidate to a foregone conclusion. I usually avoid questions like, “Tell me at least three ways to test this,” unless I’m sure that everyone can discover at least three ways. Some people stop at three answers even though I’ve asked for more. Remember, people are nervous during interviews and may not be thinking clearly. I sometimes write out the audition request on a piece of paper so the candidate can have a written copy of the question(s) he or she can refer back to.

Test the Audition with Your Current Staff
Once I develop the audition, I test it with the current staff. I once developed an audition only I could succeed at—a terrible audition for other people. Not only do I consider the technical part of the audition, I make sure the audition can be completed in the time allotted, and that I have time to review the results of the audition with the candidate.

An audition is not a final exam for a graduate-level course; it is a technique that reveals how a person works. It doesn’t need to be hard—but it must reflect your working environment.

Summary
Auditions help you see a candidate in action. Whether you hear a candidate’s problem-solving process during a phone screen or watch the candidate at work, you’ll gather a richer perspective of the candidate’s capabilities and how easily the candidate can adapt to your working environment.

Leave a Reply