What worried me was that some candidates got the answer to my question instantaneously. Almost no thought needed. Oh, so you want to accomplish X? How about a scheme that does Y? Other candidates needed to think about it for some time. But this one didn't get it even after about 15 minutes. Are the ones that get it instantaneously really so dramatically better? I knew from experience that many who came up with the answer fast didn't do well on the next part of the question, or overall in the rest of their interviews.
As the interviewee continued to metaphorically bang his head on the wall of the question, I realized that this part of the question was meaningless. I couldn't say for sure that a great performance on it would indicate a great engineer, or a poor performance would indicate a poor engineer. Worse, my candidate had to struggle without making progress for quite a while. He wasn't feeling good about it.
I was ultimately wasting time. 15 minutes spent determining that the interviewee can't answer a specific issue is 15 minutes that only yields a small amount of information. Either the interviewer realized the answer or not, so I really only got that binary signal.
I was ultimately wasting time. 15 minutes spent determining that the interviewee can't answer a specific issue is 15 minutes that only yields a small amount of information. Either the interviewer realized the answer or not, so I really only got that binary signal.
At that time I had another question in my repertoire, which was a simple coding exercise, significantly more complicated the FizzBuzz, but less complicated than writing merge sort. Pretty much the exact opposite of this type of question. Anyone that can write code could do it. But it turned out that performance on it was still variable, which was good, and I never got the feeling that performance was random. Also, I was able to get all sort of signals: how fast the candidate wrote their code, if they remembered details about their language, if they made mistakes, if they remembered to test their code, etc.
After this bad interview, I switched almost exclusively to this simple coding exercise. After I made the switch, I never had a bad interview again. Candidates always made progress on this question. They almost always finished the implementation. They may have had to think about some parts, but they always left thinking they did a reasonable job. And I never had to sit through and puzzle out how to drop hints to solve some algorithmic problem that the candidate is just not grasping.
With everyone pretty much able to finish the question, I judge candidates on the following:
If you are thinking of asking a coding question, here are my recommendations, based on an interview time of 45 minutes:
After this bad interview, I switched almost exclusively to this simple coding exercise. After I made the switch, I never had a bad interview again. Candidates always made progress on this question. They almost always finished the implementation. They may have had to think about some parts, but they always left thinking they did a reasonable job. And I never had to sit through and puzzle out how to drop hints to solve some algorithmic problem that the candidate is just not grasping.
With everyone pretty much able to finish the question, I judge candidates on the following:
- How quickly were they able to finish the code?
- How correct is the code?
- After writing the code, did they go through the code to test it against some sample data?
- How familiar do they seem to be with the language they are writing in?
- How elegant is the code?
- Is the code efficient?
Additionally, I'd ask a algorithmic question or two after this, if we have time, to see how they can speed up the code.
Usually, the answers to these are pretty well correlated. If you can code elegantly, then you can usually code quickly, which them means you will have time to test and correct the code. At the end, I think I have a good judgement of how decent a coder the person is. If the candidate is a good coder, I almost always recommend hiring.
If you are thinking of asking a coding question, here are my recommendations, based on an interview time of 45 minutes:
- Make sure you can write the code from scratch in about 15 minutes. That doesn't sound like a lot of time, but you already know how to solve it, presumably, or else you wouldn't be asking the question. All candidates, good or bad, take longer than you may think to code something up.
- Have follow up questions that are algorithmic in nature. What is the running time of the code? How you can make it faster? Could you parallelize it? This is all for when the coding is finished. If the coding never finishes, don't ask these questions.
- Insist on real code, not psuedocode. Let the candidate choose the language they are most comfortable in.
- If you don't have a good coding question, look at the last few bugs you fixed. Could one of them be an algorithm that could be simplified and turned into a coding exercise? The existence of a bug is proof that the algorithm isn't too simple.
Judging a candidate can't be done through coding questions alone. Algorithmic and design questions are necessary too. But I'd ask coding questions first, since they seem to offer the most solid and reliable feedback.
3 comments:
I've done hundreds of interviews and found that questions which have short, perfect answers are basically bad. If the candidate gets it quickly then you've learned very little about them.
I try not to design interview questions that have 'perfect' answers and always leave room for discussion at the edges. One great source of feedback about a candidate comes when you start talking about API design. Get them to really think about use cases and making their API useful to the caller (memory usage, error handling etc...) there's always plenty of discussion there once they've got the code out of the way.
Agreed, David. API design questions are also great, and test out different skills than coding.
Right, so I start with the fairly simple coding question and if we get through that we talk about its API.
The added bonus of this approach is that if we only do 3/4 of the coding, the candidate goes away thinking they did reasonably well in the interview (even though they didn't really). That way you've avoid breaking them for the next interviewer.
Post a Comment