Conducting Consistent Engineering Interviews: A Framework for Success
By Seth Black Updated September 27, 2024
A Framework for Software Engineering Interviews
After more than two decades as an engineer and over a decade as a hiring manager I've experienced my fair share of terrible engineering interviews. From quizzes to whiteboarding, take-home tests to awkward real-time coding exercises; I feel as though every startup I've worked at has had the exact same growing pains. To help out, and keep myself from going crazy, I've developed this framework for interviewing and assessing engineering candidates. I've specifically positioned this as a framework to help guide, and not a methodology to prescribe. My genuine hope is that y'all can take this and use it to be better interviewers and better prepared candidates.
Stage Zero: Housekeeping
There are a few things that I recommend keeping top-of-mind while conducting an interview. While not exhaustive, this list covers topics that are important to me and help keep the process efficient while making room for a more diverse candidate pool.
- Maintain adequate self-awareness. I have said my fair share of dumb things during interviews (If you feel like you have not may I strongly suggest seeking an outside opinion). Instead of struggling to speak the perfect words one-hundred percent of the time I acknowledge that I'm not perfect, I keep an open mind, aim for objectivity, empathize with the candidate, and I quickly apologize whenever I misspeak.
- Fill out the scorecard and take notes in real-time. I oppose note-taking and putting words in candidates' mouths after-the-fact. The best time to document something is right when it happens. Every second that passes gives our minds and our own biases time to obscure the moment.
- Seek alignment to vision instead of a “culture-fit”. Culture, by its very nature, is not very diverse; however many different paths can be used to achieve a common vision. Having a strong vision that is inspirational and empowering will beat out a vague motto and dubious culture every time.
- Include enough people in the process without overwhelming any one person. Respect your candidates' time, your colleagues' time, and your own time. Seventeen billion rounds of panel interviews with three to five people on each panel is stressful and expensive. Leaving one person in charge of the entire hiring process is stressful and ineffective.
Stage One: Resume Parsing
In Computer Software there are few managerial tasks as brutal and mundane as manually scanning through a folder full of resumes. The very idea of a formatted resume is quite strange to me. Computers are able to help diagnose complex medical conditions, have unpacked the human genome, and are even able to recommend - with almost perfect precision - the next show we will binge watch. With all of this amazing technology, we still spend copious amounts of time formatting, editing, revising, and critiquing a text document that will be shoved into the depths of this year's most popular HR Recruiting software. At the bare minimum, as a human OCR machine, what is it that I am specifically looking for?
- A solid cover letter. Nothing fancy or too long. A few short sentences telling me who you are and why you want to work with me, my team, and my company. Are you passionate about robotics? Machine Learning? Python? Sweet! Help me know these things, and quickly and succinctly as possible.
- A resume that displays overlap with the job description. I'm not looking for exact word matches. No, I'm looking for proof that you have the requirements necessary to be a Computer Programmer as well as enough transferable knowledge that I'm not concerned about spending the first six months training you from scratch.
Stage Two: Phone Screen
I love meeting new people, so the first phone screen is one of my favorite parts of the hiring process. There really is no magic with early phone screens. Are you a human? Check. Do you understand what job you're applying for? Check. I typically do a quick resume review to fill in any gaps I may have, review the job description to answer any questions the candidate may have, and chat a bit about the company and the general hiring process.
Stage Three: Engineering Interview
The Engineering Interview is the part of the process that seems to challenge us the most. I generally feel like take-home tests waste too much time. Whiteboard coding appeals to a very specific type of person - I can only speak for myself but I have not once shipped whiteboard code to production. Panel interviews with tricky word-problems tend to feel more like interrogations. Sharing a screen while fumbling through a problem is quite unnatural and uncomfortable. I have overcome this challenge - with a fair degree of success - with a two-part interview. These two parts can be combined into a single 30-45 minute interview, or broken up into multiple sessions depending on your needs.
Part One: Competency Scorecard
I have assembled a competency scorecard with general categories and conversation starters for each major topic. I have provided an example completed scorecard below. The expected outcome is not for every candidate to perfectly answer each topic in the scorecard, but to be used as an overall indicator of the candidate's skills. It's also extremely important to note that if the interviewer is not familiar with any of the topics in the scorecard they should be strongly encouraged to review the subject material, or possibly move to a different department - preferably in a competitor's organization.
Under the Python Language Skills heading I have the topic Logging. I may start a conversation with something along the lines of, “I tend to run into code bases where some modules rely on print statements and others rely on the logging module. What is your preferred method for reconciling the two strategies?” If you're solid enough in Python you'll know that question has a fairly straightforward answer, yet still leaves plenty of room for discussion. I feel this method gives the interviewer an easy handle to score with and keeps things moving along when a candidate is not well versed in any particular subject. I use simple ballot boxes to check off skills, topics, and subjects as they are discussed during the interview.
| Python Language Skills | AWS Skills |
| ☐ Core Data Types | ☐ Well Architected Framework |
| ☐ Logging | ☐ Broad Service Knowledge |
| ☐ Debugging | ☐ Different RDS Database Types |
| ☐ List Comprehension | ☐ S3 and Glacier |
| ☐ Generators | ☐ EC2's relationship to other services |
| ☐ Exception Handling | ☐ Lambda |
| ☐ PEP8 | ☐ Glue |
Part Two: The Code Review
I've discovered that giving Computer Programmers the freedom to judge others' work is a really easy way to expose their abilities and shortcomings. The collaborative nature of code reviews removes a lot of fear and anxiety, and code reviews set clear boundaries that prevent long awkward pauses. I try to keep a small, fresh rotation of small Python scripts that accomplish tasks such as list and modify files in an AWS S3 bucket, parse JSON files, or create a socket-based client-server. Recently I saw a Principal Engineer use a candidate's own code, which I thought was interesting.
Consolidation and Decision
From a technical perspective, I feel like I am able to gather enough information through resume parsing, phone screen, and technical interview to confidently determine whether or not the candidate has the technical skills necessary to excel at the job they've applied for. I very rarely need more discussion after filling out the Skills Matrix and going over the code review.
What about things like soft-skills? If I'm unsure, I typically just out-right ask candidates. What if they lie!? Well, that's part of being in a situation where you're required to interact with other people. Are there dishonest people in the world? Yes. Would they lie about being dishonest? Definitely. Do I approach each new situation assuming every new person has bad intentions? No. Do I hold everyone around me accountable for their actions and words? Absolutely! My wife worked for a chef who had my favorite standard for his employees: "I'm paying you to be the person that I interviewed and hired."
As engineers, we tend to hold ourselves to unattainably high standards: one missing colon will prevent a Python script from running, an extra comma breaks the JSON parser, one wrong answer and you've blown the technical interview. By bringing other people into the mix we greatly reduce our ability to develop a perfect system, and that's okay. My hope is that you can take this framework and use it to develop a strong and consistent hiring methodology that lines up great candidates with great organizations.
Good luck and happy hiring!
-Sethers
