Many many years ago when I was managing software development at a small startup, I struggled to find developers who could develop software. The candidates had the right skills on their resumes. The candidates answered the questions we asked. But they just weren’t up to the daily responsibility of writing software. Frustrating. What was I missing? The answer came to me in a book I was reading. It changed the way we interviewed developers. It also changed the way I do interviews today.
My Interviewing Epiphany: How Do You Hire a Juggler?
The book was Peopleware: Productive Project and Teams by Tom DeMarco and Timothy Lister. It’s a little dated today. It preceded the agile revolution and the significant changes to the way we develop software and manage projects. It also anticipated that revolution and incorporated several practices we take for granted in the 21st century.
In one chapter, DeMarco and Lister posed an interesting analogy. If you were to hire a juggler for your circus the way we hired engineers, they suggested, your hiring process would likely include an exchange like this:
Circus Manager: How long have you been juggling?
DeMarco, T., & Lister, T. R. (2014). Peopleware: productive projects and teams. Upper Saddle River, NJ: Addison-Wesley 103.
Candidate: Oh, about six years.
Manager: Can you handle three balls, four balls, and five balls?
Candidate: Yes, yes, and yes.
Manager: Do you work with flaming objects?
Candidate: Sure.
Manager: . . . knives, axes, open cigar boxes, floppy hats?
Candidate: I can juggle anything.
Manager: Do you have a line of funny patter that goes with your juggling?
Candidate: It’s hilarious.
Manager: Well, that sounds fine. I guess you’re hired.
Candidate: Umm . . . Don’t you want to see me juggle?
Manager: Gee, I never thought of that.
That was an epiphany moment. I was hiring jugglers, but I wasn’t asking them to juggle.
DeMarco and Lister explore three different approaches to seeing examples of the work: aptitude tests, portfolios, and auditions.
They mention and promptly dismiss aptitude tests. Not that tests themselves are necessarily bad. A test can be a great way to determine if a candidate has the knowledge to perform a job. The problem is that tests are good at testing what people know today and you are hiring for tomorrow. This is important for knowledge workers who will need the knowledge and skills to solve the next set of problems. There are additional problems today where many tests are shown to have a discriminatory effect.
Portfolios of past work are common in some professions, especially in the arts and design fields. My design students always have portfolios of their designs. Portfolios are less common in more technical fields such as programming and analysis. You can learn a great deal from looking at something that a candidate did previously. There are two challenges with portfolios. First, unlike a portfolio of fashion designs or websites or photographs, it is difficult to get a snapshot of work product that includes context. I could show you the business case that I developed for a recent project. But you would need time to digest and understand it in context. Another challenge is the proprietary nature of knowledge work. Could I, as a candidate, really show you that business case? Should you even be asking for it?
Auditions are an excellent mechanism to assess a candidate’s capabilities. Not only do you gain insight into their technical skills, but you also see how they handle the social dynamics work. This is the direction that I went in my revised approach to interviewing.
My First Juggler Interviews
Armed with what I had read, I sat down and made a plan. Instead of hours of behavioral interviews where asked developers about developing, I would focus the interview on two audition components: Programming and presentation.
Programming
I gave candidates a simple, but non-trivial programming task. It was a very specific calculation in our industry and described completely in less than a page. I provided them with a written description of the calculation, a quiet room, a computer with the development tools, a reference library, and a diagram of the database structure used by our application. I gave them two hours to write a function that would perform the calculation and return the result. They asked whatever questions they wanted before starting and were told that I would return in an hour if they had questions. I also told them I did not expect them to complete the task, but wanted to see how they approached it.
This was the most important part of the new interview process. It reflected exactly the work that our developers had to do daily. I provided an opportunity to assess not only the candidate’s ability to write software but also addressed their ability to work under pressure, to solve problems, and to ask questions. None of the candidates finished the task. All but one had something to show for their efforts. The best had a structured approach that broke the problem down and had significant components working by the end of the exercise.
Presentation
I also asked candidates to prepare a 10-15 minute presentation on a business or technical subject of their choice. The audience for the presentation would be our development team. They should expect an additional five minutes of questions.
The presentation was to assess another import requirement. Could the candidate communicate complex information to other developers? We also used this to assess ability to work unsupervised to a deadline and the quality of their work product. We used a less formal presentation environment — everyone sitting around a conference room table — and the Q&A session usually ended up as a discussion about the topic. So we assessed the candidate’s ability to function as a member of the team.
We all learned a great deal about a variety of topics in the course of those interviews. Some topics were technical — technical challenges resolved or new technologies explores. Some topics were about the business domains in which the candidates worked — business problems in home security, healthcare, or insurance. The presentations themselves varied. Some candidates even brought handouts. One skipped the “presentation” and just talked.
Hiring My First Juggler
We conducted “auditions” with five different candidates. A sixth, when told what the interview would entail, opted not to continue. Of the five, we eliminated three as not having met our criteria in the two tasks. We had a spirited discussion about the other two. (I’ll talk more about how I manage those discussions and reach a decision in the next article). When we made the offer to our final choice, he accepted immediately. He also commented to me he really enjoyed the interview experience. He specially noted that it had given him a great sense of who we were, what we did, and how we worked. That was a factor of the audition approach that I hadn’t thought of. He also turned out to be one of our best employees in the long run. Not a bad experiment.
How I Interview Jugglers Today
Decades later, I’ve adapted my audition approach somewhat. This is a result hiring fewer developers and more analysts and project managers. I replace the programming task with something else that speaks to the requirements of their jobs. I’m also in different environments where I have less control over the interview process. I need to fit the audition concept into a different set of constraints. But I can still apply the concepts that shaped my earlier approaches to hiring jugglers.
POD — Problem of the Day
Alas, I no longer have a calculation defined by a government regulatory agency that I can hand over to a candidate as a programming task. Even if I did, what would a project manager or a business analyst do with it? But I have a plethora of problems and challenges that I am dealing with daily that can provide insight into how a candidate would function in our environment. As I prepare for interviews, I give thought to the job we are filling and look for challenges we have faced in recent weeks that can I can encapsulate into brief exercises for candidates.
These exercises, which I refer to as Problem of the Day questions, have certain requirements themselves:
- I need to explain the problem in less than five minutes;
- I need to provide a document of only one page to summarize the problem and the expectations;
- The problem can require no additional information or material other than what I can provide on that summary sheet or through questions the candidate might ask;
- There needs to be no clear right or wrong answer to the problem;
- There can be better or worse approaches to solving the problem.
That’s a lot of constraints. But those constraints make it easier to define the problems. Here are two Problem of the Day questions that I’ve used recently:
One POD I’ve used with project managers is a stakeholder analysis problem. I provided a brief description of a real project. This description included the business objectives and an outline of how we approached the problem. I included a list of potential stakeholders in the organization. The list included first names, titles, and one-sentence descriptions of responsibilities. I used no actual names or titles in the exercise, but the job descriptions were in keeping with those of our organization. I asked the candidate to review the information, prioritize the stakeholders based on their potential impact on project success, and identify the key stakeholders for the project. I allowed five minutes for questions at the beginning of the exercise. The candidates had an additional 15 minutes to complete their analysis.
For business analysts, I like business case problems. I’ll start with a brief description of a project, including business objectives and approach. I might include the stakeholder list. But instead of a stakeholder analysis, I ask for a convincing business case to complete the project. In this variation, I’ll allow five minutes for questions at the beginning and return in ten minutes to answer questions. The candidate has another ten minutes to complete the exercise.
Teach Me About…
Nor have I had much opportunity to coordinate presentations on business or technical topics to an interview panel. That requires a level of coordination of already crowded schedules that has proven to be a challenge. It doesn’t mean that I have to abandon the presentation model itself. With a little preparation, anything in a candidate’s background can become the subject of an impromptu presentation.
I call these “Teach Me About” questions. While preparing for the interview, I will identify one or two items from the candidate’s background I would like to learn more about. Then, during the interview, I inform the candidate that I would like to learn more about one of them. Rather than ask questions, I’d like her to take about ten minutes to prepare a five minute “lesson” on the subject, with another five minutes for questions. No need to make a formal presentation. We just use the whiteboard. I leave them with a few sheets of paper for notes and leave the room. I give them the opportunity to ask questions before they start. When I return, I get a brief lesson on a business or technical topic identified on the resume.
Like the group presentation, this provides an opportunity for the candidate communicate a business or technical topic. Unlike the group presentation, there is an audience of one and the candidate doesn’t have the time to prepare in advance. But I work to keep it informal and keep the topics from the most recent experiences to ensure that they are fresh.
Grading the Audition
Yes. I have grading rubrics for both my POD and Present to Me questions. In each case, I review the problem and determine what a baseline, good, and outstanding responses will be. More precisely, what are some characteristics I see in baseline, good, and outstanding responses.
For POD exercises, baseline responses usually start with an honest attempt to solve the problem. And yes, I’ve had candidates who did not understand how to approach these problems. Good responses include some structured approach to solving the problem. In most cases, there are several techniques or frameworks a candidate could use. Using one of those definitely checks the “good” box, as does coming up with a new approach if there is some structure to it. Good responses also include good questions of me. Questions that show that the candidate understands the nature of the problem and can recognize what information I excluded are hallmarks of a good response. Outstanding responses bring that something extra. Maybe a novel approach or particularly insightful questions.
Baseline responses for Teach Me About questions also start with an honest attempt to convey the information and some level of organization of the material. A rambling monologue about the topic is not a baseline response. Good responses have more structure and break the topic down more effectively. They also include more of a dialog to ensure that I’m following along. The best answers leave time for questions. I also look for someone who, when given the opportunity to ask questions beforehand, checks my level of understanding about the topic. Another plus is checking to ensure that they know what they should cover.
Brain Teasers
Before we leave the subject of auditions, we should talk about brain teasers. I worked in management consulting in the early years of the 21st century. During that time and in that industry, brain teaser problems were very common in interviews. They come in and out of fashion today in several places. Companies like Google make them key parts of their hiring process. You know the ones I mean: Why are manhole covers round? How much would it cost to move Mt. Fuji 150 kilometers north of its current location
I’m not a fan of brain teaser questions.
Employers use them to judge a candidate’s ability to solve problems by thinking outside the box. I disagree. I see them more like riddles. They are baffling until somebody tells you how to solve them. Then they appear easy. And like assessments, they measure the wrong things. Just because you can solve an obscure brain teaser puzzle doesn’t mean that you can address the challenges that you will face daily in the work environment. Unless your job description is solving brain teaser puzzles. So instead of wildly inventive brain teaser questions, I put my faith in realistic problems taken from my current work environment and business domain. I like out-of-the-box solutions as much as the next guy. And you’ll get points for an out-of-the-box solution with me. But you get more points, and are more likely to get points, from a thorough analysis and a structured approach to solving a real-world problem.
How Will You Interview Jugglers?
You can’t hire a juggler for your circus or your child’s birthday party by asking them to tell you about a time that they juggled. Nor can you hire a knowledge worker only by asking them to tell you about a time when they solved a problem. You need to give them the opportunity to solve a problem for you. Not any problem. You need to give them the opportunity to solve a problem relevant to your industry, company, and environment.
Look carefully at the job description and then think hard about the problems you are facing today. Can you turn one of those in to a one-page problem that a candidate can try to solve in an hour during an interview? Can you use information on their resume to prompt an informal presentation to teach you or other members of the interview panel something? This approach to interviewing will give you better insight into a candidate’s abilities. It will also give the candidate better insight into your organization and the problems she will face there.
What are you going to include in your auditions?
Interviewing The Knowledge Worker — A Guide for Practitioners
Asking The Right Questions
Making the Decision
