Double-Blind Coding Interviews

https://www.pexels.com/photo/person-using-gray-laptop-3861964/

If you have done a coding interview at any point in the last 20+ years, you have likely been provided a problem to solve by the interviewer. Sometimes you have a dry-erase marker and a whiteboard and sometimes you have someone else’s laptop or your own machine to solve it. The problem itself is likely pulled from a place like LeetCode or, if you are lucky, it is a problem the interviewer or team has struggled with in recent history. Lately, the “take-home” problem has become more popular which allows the candidate more comfort on their own machine and less pressure by doing it on their own time. Regardless of the specifics, code interviews follow the same pattern of:

Interviewer provides you some problem they have thought about a lot and you have some much smaller time and less context to try and solve it.

More and more interview panels are accounting for unconscious and conscious biases in the interview process by taking steps like not sharing feedback until everyone has provided it, anonymizing resumes, and providing unconscious bias training for interviewers. These are all excellent, and necessary, steps to improve the selection process. However, there are still fundamental aspects of the way we conduct coding interviews in the industry that will always introduce bias.

In a blind or blinded experiment, information which may influence the participants of the experiment is withheld until after the experiment is complete. Good blinding can reduce or eliminate experimental biases that arise from a participants’ expectations, observer’s effect on the participants, observer bias, confirmation bias, and other sources. (Wikipedia: Blinded Experiment)

Living in a pandemic the past 2+ years has introduced me to more medical studies and learning about some of the specifics to how these tests are structured. With that framing in my head, I wondered why every coding interview I’ve ever done or administered was set up as a single-blinded experiment. The interviewer has some problem they have asked dozens of people before and seen dozens of solutions to already. The candidate has to try to work through this on their own by gathering context and trying to account for every edge case and possible integration this small piece of code or architecture would need to cover. On software engineering teams, we work together to solve problems that potentially none of us have the answer to already. So why don’t we do that during the interview process?

I started asking around my network and posted a general question to Twitter and (not?) surprisingly, this wasn’t a thing anywhere that I could find. The closest example was having the candidate work on an *actual* problem your team is working on and possibly blocked on. This is a great way to give the candidate a sense of the problems they will be working on, but still puts them in a position where they have much less context about the system, its limitations, and its rules than the interviewer does. I’ve also heard feedback from candidates that these sessions feel like the interviewer is using them as a rubber duck for their own problems and doesn’t give them and effective way to show their abilities. If they don’t have specific domain knowledge or enough context about your system, it can be very difficult and sometimes provide false positives or negatives because of it.

Administering Double-Blind Coding Interviews

So, what can we do about it? Are double-blind interviews even possible? What would that even look like from both the interviewer and the candidate side? At a high-level, it’s quite simple: Have the interviewer and the candidate solve a problem together that neither of them have seen before. This evens out the context imbalance and makes the solution a shared output where you can evaluate the candidates collaboration, technical knowledge, problem solving, architectural decisions, and system design. Depending on the experience of the people and the role, you can even gain signals for leadership, responding to push-back, “grit”, mentoring, and what it is generally like to work with the person.

For a typical 1-hour technical screen interview over video call, here is a structure I’ve started to put together:

Roles: Hiring Manager, Interviewer, Candidate

1. [0:00] Hiring Manager, Interviewer, and the Candidate join the call.
2. [0:00–0:05] Brief introductions for all 3 participants.
3. [0:05–0:08] Hiring Manager gives a quick overview of the double-blind format (should also provide this to the Candidate beforehand) and introduces a shared coding environment (Ex: Codility or CoderPad).
4. [0:08–0:10] Hiring Manager introduces the problem, answers initial questions, and leaves call. Note: Hiring manager should be available via chat for other follow-up questions or help with issues
5. [0:10–0:55] Interviewer and the Candidate work through the problem together. Both should be encouraged to look things up as necessary and work under the assumption that others will have to maintain this code in the future — use coherent naming conventions, tests, styles, and comments to help communicate clarity.
6. [0:55–1:00] Hiring Manager joins back into the call and helps answer any questions about the role, company, or in general that the Candidate has. As much as possible, the Hiring Manager should not look at the solution or discuss the problem itself.
7. [Post-Technical Screen] Interviewer saves the code and writes up their feedback on the candidate with their produced code based on their time and the specific areas they were trying to gain insight into.
8. [Debrief Meeting] The Interviewer reviews their time with the candidate and the solution itself for the hiring team to help inform the hiring decision.

This approach isn’t much different than a “typical” technical screen. The only differences are: 1) The interviewer doesn’t know the problem or optimal solution and 2) The Hiring Manager has 15 additional minutes allocated for interview loops. This example was framed in a technical screen format but it would be fairly straight forward to apply this to onsite rounds as well.

Potential Issues

How can you ensure interviewers haven’t seen problems before?
You can’t be 100% sure, but you can make this a significantly rare edge case with some planning. First, generate a large pool of questions (20–30+ per role) that are relevant and abstract enough to provide at least an hour of effort to solve. Second, assign some fun names to the problems so you can double-check if the interviewer as done this one before. If they have, find a different problem or a different interviewer. Third, rotate the interviewers that facilitate the double-blind interviews. This seems to be a good way for your engineers to ramp-up on interviewing in general since there isn’t much administration or strategy to the interview structure, it’s approximately the same thing they would be doing during the day anyways.

Shouldn’t all the questions be the same for each candidate to make evaluations equivalent?
This is a tough one. On one side, yes, each candidate should have the equivalent experience and expectation so that they can be compared fairly to other candidates. On the other side, this interview style is screening for non-technical characteristics like collaboration and ability to take feedback so you will never have the same experience when humans with emotions and outside pressures from life are involved. Additionally, you could provide questions that appeal to the candidates background like Machine Learning or Kubernetes and help set them up for success instead of forcing binary tree traversals or string manipulation on them that may or may not be relevant to their background. I would love to see some research or empirical data around this if anyone has it.

I’m a manager and hire a lot, 15 more minutes of meetings for each candidate adds up really fast and wouldn’t be possible
I hear you, managers are typically in a ton of meetings! As long as the structure of the interview is clear and the person has context to the parameters of the question, it doesn’t have to be the hiring manager that plays the role. Recruiters, senior members of the team, or even other managers could play this role. I feel that the hiring managers presence helps the overall experience of the candidate and ensures continuity through the process but if schedules don’t work out, replacements are totally acceptable.

What if the candidate just lets the interviewer do all the work?
It is a shared responsibility to solve the problem, but the interviewer should be actively engaging the candidates opinions, code, and problem solving approaches. If the candidate isn’t able or willing to contribute to the solution, that gives a signal** to how they would interact with the team in the future. With the explicit freedom to look things up, it should mitigate the amount of uncertainty in the problem and present a scenario quite similar to what a typical problem and day would be like with the team.

**There are many things that could contribute to this, so it is merely a single signal that should be taken into account with everything else in the loop.

What other issues or benefits do you see here? I’d love to hear about any efforts to set this up or conduct interviews in a similar way. I’m just starting to experiment a little with it in my hiring so hopefully will have some feedback soon!