When is a year of experience not a year of experience? When that experience doesn’t match your needs. Take my friend Zack for instance. A VP of Development, he recently came to me with a problem.
“We hire the best people money can buy,” he said. “They have degrees and tons of experience. Why can't we finish our projects on time?”
Zack wanted his organization to crank out 4-6 releases per year. So far this year, they’ve released two—both of them were late. “Something has to change. Maybe it’s my management. Maybe I expect too much.”
“Maybe it’s who you’re hiring,” I suggested.
“They are smart people. They should be able to perform well.”
“Perhaps you aren’t hiring people with the experience and skills you really need. Let’s look at that. Can you think of any delays you’ve had that you can attribute to lack of skill?”
Zack considered that for a moment. “Well, there’s Mike. I hired him a few months ago. He has five years experience developing: low and mid-level design, coding, fixing. So silly me, I figure he knows what he’s doing. But I was wrong. Someone with his experience should know that you hard-code as little as possible in a function. And, if you’re developing several similar functions, you group the common pieces into subroutines or classes, depending on your language. Right?”
“Sure,” I replied.
“Well, Mike was implementing several serial drivers and forgot to add a key piece of functionality into the first driver. The tester reported the problem to Mike. When the tester received the next build, he first checked the original serial driver for the problem, and it was fixed. But when he checked the next driver for the same problem—sure enough, the original problem was still in second driver.”
“That will certainly cut down on your productivity.”
“You betcha! Oh, and then there’s Minerva. I hired her 4 months ago. She’s a tester with over 10 years of experience. Great hire, right? But listen to this. Minerva thought the only technique to test a simple user interface was to walk through the entire GUI, singly changing each parameter. It took 30 work-hours to test her product’s GUI that way. So Maxine, who sits in that cube over there, and only has 3 years of experience, mind you, had to show her how to speed things up. See, Maxine had read about two-way combination testing—you know, the test reduction technique that would reduce the total number of necessary tests to a fraction of the original number.”
“Zack, it sounds like both Mike and Minerva were missing some functional skills that your project required, even with all of their years of experience. There are questions you can ask to help you find that out before you hire them. You have to probe for specific skills you feel are important. We can work on that. Who else do you have?”
Zack was on a roll now. “And then there’s Mackenzie, a project manager with 6 years experience. He transferred in from another division. In his last position, he successfully managed five small projects using a spiral lifecycle. I assigned him to manage a staged-delivery lifecycle project with more people and a longer total project time.
During the initial project planning, Mackenzie tried to organize the schedule so that the project team could modify the entire product. What he should have done was take enough time at the beginning to define enough architecture and interfaces to allow several sub-project teams to work in parallel. One of the developers had to take Mackenzie aside and explain how a staged-delivery lifecycle worked.”
“ Well, Zack, not every project manager with 6 years of experience has had the opportunity to plan projects in multiple ways. Even if the project manager understands the theory of different lifecycles, the project manager may not be able to apply that theory to your project. It sounds like Mackenzie didn’t know about any lifecycles other than the spiral lifecycle. Perhaps Mackenzie’s projects were organized so similarly that his experiences with multiple lifecycles were not accumulative. One way to detect the repetition of experience is to ask a candidate, ‘What was different between these two projects?’ and then ask, ‘How did those differences affect how you managed the projects?’ When you listen for the answers, hear if the candidate changed their work in reaction to the project, or if they tried to bend the project to fit how they work. The people with multiple years of differing experience will adapt their approaches to different work. The people with the same year of experience repeated multiple times will bend the project to fit their work.”
“Wow. That’s a really good idea. Listen, I need to hire someone new. Can you help me work out some questions to ask?”
“Before we can come up with questions, we need to first define what a successful candidate for your position should know.”
Zack’s problem is one of ill-defined expectations and incomplete evaluations. He was relying on college degrees and years of experience to gauge a candidate’s fit for his organization. But, you can’t assume that because a candidate has six years of experience, the candidate knows what you learned in six years or what a person in your organization typically learns in six years. What a person learns in six years depends on the types and variety of assignments, and perhaps on how many different organizations the candidate has worked for. As a hiring manager or as part of the hiring team, it's your job to assess the candidate’s experience against the experience you need.
Hiring the right person is a three-step process.
1. Define the experience you require in a candidate—What do you need?
2. Ask how the candidate learned their current experiences—What do they have?
3. Evaluate what the candidate’s experience against your requirements—Do they have what you need?
What Experience Do You Need?
When you write a job description, consider the multiple dimensions of experience: functional skill, product domain, tools and technology, and industry experience.
We acquire functional skill experience in school or classes, when we learn all the details of the function of software engineering that we perform (user interface designer, tester, project manager, reliability developer, and so on). We develop our functional skill experience at work: designing, developing, testing, supporting products, or while managing projects or people.
Many organizations believe they’ve covered functional skills by requiring a bachelor’s degree for technical positions, Unfortunately, a degree is no indication that the candidate has learned your expected functional skills or how to apply them. Candidates with technical degrees in computer science, software engineering, or math, may have learned how to perform these activities in college, but may not have ever really applied them in real life.
It is not until people work on a real project with other people, deliver some of their work product to the rest of the team, and have that work product reviewed explicitly or implicitly, that they finally have enough feedback to learn what they know and what they don’t know. When you recruit candidates based on years of experience, you’re looking for the application of functional skill experience beyond formal education.
Zack wanted to know what sorts of questions he could have asked to uncover whether his candidates’ functional skills matched his needs.
• He could have asked Mike, “How do you track your problems—the errors you create?” Whether you use engineering logs, journals, or some other method, simply keeping track of errors helps you learn from them. Finding out Mike didn’t do this at all might have been a red flag for Zack.
• He could have asked Minerva, “Tell me about an instance when you saved time on a project,” or, “Tell me about a time you found more problems than you expected on a project.” Answers to those questions would likely have told him whether or not she was experienced in using alternate testing techniques.
• He could have asked Mackenzie, “What was the difference between Project A and Project B? How did those differences affect how you managed the project?” Zack could have learned whether Mackenzie tended to change his work for each project, or tried to bend the project to fit how he worked.
These kinds of questions explore a candidate’s problem solving skills and adaptability. The questions move beyond years of experience to how people have applied their functional skill experience. Functional skill experience is the cornerstone of how we perform our jobs, our entry to the position. Product domain experience is the next building block, how people use their functional skills for the good of the product.
Product domain experience encompasses how people understand the product and the product domain. I use three levels to product domain experience:
• Level 1 is exposure and general understanding of the product externals.
• Level 2 is the understanding of the internals of the product.
• Level 3 is the ability to relate the product internals to a person’s functional skills, and to other products, to see the similarities and the differences.
For example, if you have a multi-threaded product, one where numerous instances of the product can be run simultaneously on one machine for multiple users, then Level 1 knowledge is knowing that the product is multi-threaded. Level 2 knowledge is knowing what kicks off multiple threads. Level 3 knowledge is using the knowledge that the product is multi-threaded to develop pieces of the product appropriately, or to create tests that test the product differently than if the product was single-threaded.
Rating Product Domain Experience
At Level 1, people understand how the product interacts with the users, or from black box level, how the product interfaces work. Great tech support people and black box testers generally acquire this expertise quickly.
At Level 2, people understand how this product works on the inside, the architecture of the product. We expect developers have this expertise or can acquire it by reading the code. Writers can acquire this expertise, depending on the kind of documentation they write. Testers and tech support staff can learn this kind of expertise if they can read the product internals, or if they are taught the internals. Technical project managers may develop this understanding if they understand how systems are architected and developed.
At Level 3, people understand the specifics of how this product works, and can understand why they would use a certain functional skill to develop or test the product. Senior developers, senior testers, senior designers, and architects are likely to have this expertise. It’s easier to obtain this level of expertise if you read and write code. However, just knowing how to read and write code doesn’t ensure you have level 3 experience: the ability to relate your functional skills to the multiple nuances of the product domain. And, even if you don’t have code reading and writing ability, you can still acquire level 3 domain experience.
So how can you tell if a candidate has product domain experience if they’ve never worked on your product? Look for ways a candidate’s experience matches your domain expertise needs with questions like these:
• Tell me about a time when you’ve considered performance as an issue in your development.
• Tell me about a time when you’ve dealt with testing a product where you only had access to the GUI.
Functional skill experience is what you learned from formal classes or on your own. Domain experts apply their functional skills in new and innovative ways to improve the product. Tools and technology experience, in addition to industry experience, are the context builders for how you use functional skills and domain experience at work.
Tools and technology experience is knowing and using different languages, compilers, testing tools, defect tracking tools, databases, and so on. Tools and technology are specific to the your product environment and organization.
To determine if a person has mastery with a certain tool, ask detailed questions that someone with mastery in that tool should know. For example, if you were interviewing a project manager who claimed experience with a project-scheduling tool, you might ask, “How have you dealt with rolling up sub-project schedules with such-and-such a tool?”
Don’t confuse technology expertise with functional skills. For example, if your job requires you to know Unix or Windows internals, that’s technology experience. If you just needed to know about how operating systems work, that would be functional skill experience. In the same way, being a certified database administrator (technology expertise) doesn’t mean you have any idea how to change a schema (functional skill expertise).
Sometimes functional experience and technology are so intertwined that you must probe technology skills to learn about functional skills. For example, a developer may claim to know object-oriented programming. His only experience with it, however, has been in C++. Your position requires Java experience. Does the developer have the object-oriented expertise you need? It’s hard to say, so it’s time to ask more questions. You’ll want to know: How easily does the developer learn new languages? And what exactly is his experience with OO? Here are some questions you could ask:
• Tell me about all the languages you’ve learned. (Listen for assembly languages, procedural languages, and OO languages.)
• How did you learn C (or other procedural language)? When did you know you were proficient in the language?
• How was learning C++ different from learning C? (Listen for how the developer moved from a procedural language to an OO language.)
• Tell me about a time when you used inheritance (or recursion or some other technique) from your C++ classes. (Ask about a specific C++ skill that you expect the developer to know that’s applicable to Java.)
You may hear answers such as: “C and C++ were the similar. I just had the standard pointer kinds of problems with C++,” or “There was no difference learning the second language, just how I used the compiler,” or “I didn’t need to use inheritance.” This answer reveals that the developer only knows how to use the C++ compiler (tools and technology experience) rather than understanding how OO programming works (functional skill). That’s not the object-oriented experience you need.
Similarly, if the candidate has used an automated test tool, but is only able to use the capture-replay part of the tool, does he have the test tool experience you need? It depends on your environment. Environment will also define how important industry experience is for the candidate.
Industry experience is knowing the industries in which you’ve worked: the kinds of people likely to buy these systems, and the general expectations of the customers and any other interested people. Are your customers the same people as your users? In some industries, telecommunications for example, the people who buy the systems (the customers) are not the same as the users (the people who own the phones and buy the services).
In regulated industries, such as aviation or pharmaceutical industries, the regulators are as much a part of the user base as the people who actually use the systems at work. Once you’ve worked in a particular industry, you learn how the industry works, what the customers expect, and what some of the users expect. Industry experience is helpful when deciding between alternate designs, or attempting to make the implicit requirements explicit, or when choosing which kinds of tests to run, based on risk.
What experience does the candidate have?
Once you know the kinds of experience you’re looking for, ask the candidates how they acquired that experience. When you ask these questions, you’ll learn if they have one year of experience multiple times or multiple years of experience.
You can choose many ways to ask these questions: closed and open-ended questions, behavior-description questions, and hypothetical questions are the most common. In addition, you can use rhetorical questions, meta-questions and auditions in your interviews. If you’re not sure which interview question type to use, see the sidebar entitled “Mixing it Up” for some guidelines.
Remember Zack? Zack decided that he needed people with substantial functional skill in test techniques, people who could acquire level 2 product domain expertise. He wasn’t worried about tool experience, or industry experience. Here are some of the questions we developed:
“On this last project, what test techniques did you use?” (This is a closed question eliciting the basic information.)
“Which techniques found the most defects?” (This is an open-ended question asking the candidate for their evaluation of their work.)
“Which test techniques did you use under which circumstances?” (Another open-ended question asking for a different evaluation.)
“Were there test techniques you wanted to use, but didn’t? (A closed question looking for information.) “What were they and why didn’t you use them?” (An open-ended question that asks for more evaluation.)
“What would you do if we asked you to test this product?” (A hypothetical question, to understand how the candidate thinks. And, this question gives you an opening for an audition.)
“Let me show you this part of the product.” (Zack gave a quick demo.) “We’re looking to see how you test. Here is a stack of blank bug report templates. Take 15 minutes and plan how you would test this part of the product. If you see problems along the way, fill out a bug report.” (An audition, to see how the candidate plans the testing and, with any luck, how the candidate writes bug reports.)
Zack is now asking questions that encourage candidates to tell stories, and questions that allow him to continue on a tangent, if that tangent is interesting. These open-ended and hypothetical questions help him have a conversation with a candidate.
When you interview candidates, make sure you’re also using a variety of interview questions. I focus on behavior-description questions, with a few closed questions and hypothetical questions thrown in. I use the closed questions to make sure I have the facts right. I use hypothetical questions carefully because I’m concerned I will broadcast the “correct” answer. I rarely use rhetorical questions. And, I try to use an audition once during the in-person interview for each candidate.
Whichever questions you choose, create a conversation with the candidate that can help you evaluate the candidate’s experience and capability to learn against the experience and learning capabilities you need.
Does the candidate’s experience match what you need?
Not every candidate or every employee requires significant functional skills or domain expertise skills. Nor do technical skills or industry knowledge create a perfect candidate. Review your position and define the necessary experience for the job you’re trying to fill.
When you’re looking to hire people, don’t weight past tools/technology and industry experience too heavily. It’s easy to acquire tools/technology and industry experience. Instead, focus on the functional skill and product domain experience a candidate requires for job success.
In their original hiring, Zack and his managers didn’t define the mix of functional skill and product domain experience required across the project team. Nor did Zack and his managers recognize that a candidate's ability to learn these skills was necessary in their organization. They interviewed for tools and technology expertise and Level 1 domain expertise, assuming that most people in high technology can acquire domain expertise. While it’s true that some people will perceive that every project is a learning experience and will seek new expertise, not everyone sees the need to learn more in their jobs jobs or knows how to learn on their own. Since the ability to learn new things was important to Zack, he should have looked for people who actively learn new things and apply those learnings to the job. What he ended up with were people with multiple years of only one experience.
Review the experience you and your group have. What do you need? Does your group need diverse experience or more of the same kinds of experience? Once you know, you can hire people with the appropriate number of years of experience, not just one year repeated numerous times.
And, if you think you’re in danger of repeating the same year of experience, ask yourself, what do I have to do to increase my functional skills or domain expertise? Could I take a class, read a book, and then apply my knowledge for the betterment of my organization and me? Don’t be afraid to seek out new knowledge, and find a small place in your next project to apply that knowledge.
I thank my reviewers, Elisabeth Hendrickson and Jerry Weinberg for their thoughtful review of this article.
© 2003 Johanna Rothman. This article was published in STQE, March/April 2003, pp48-53.