Interviews: Less Helpful Than We Think,or Harmful

As in many other careers, interviews play a key role in the medical profession at every stage of training and hiring. For example, a 2008 survey of 120 admissions officers from medical schools in the US and Canada (AAMC members) revealed that admissions committees consider the interview assessment the most important factor in deciding whom to accept. ¹ Like many of us, the senior author (RTS), has always placed considerable value on interviews when considering student applicants, potential residents, and candidates interested in joining our practice, faculty, or staff. However, an article in the New York Times led us to reconsider the usefulness of a process that we assumed had obvious value. This article and the research on which it was based were published by Jason Dana, Assistant Professor of Management and Marketing at the Yale School of Management. It provides insights that are both interesting and somewhat disturbing. ^2,3

As educators and employers, most of us use free-form, unstructured interviews. We believe that they give us a good idea of a person whether we are interviewing a job candidate or an applicant for medical school, residency, or another position. Dana’s observations suggest that we “typically form strong but unwarranted impressions about interviewees” ² , but that these impressions often reveal more about ourselves than the candidates. His studies highlight examples of the kinds of errors in judgment that result from job interviews.

Dana argues that his research indicates that interviews are not merely irrelevant, but moreover can be harmful, undercutting the value of other information about interviewees. In one study, he and his co-authors had subjects interview students and attempt to predict grade point averages for the following semester based on the interview, the student’s course schedule, and his/her past grade point average (GPA). ³ The researchers explained in advance that past GPA was the best predictor of future grades at their school. The students who participated in the study also were asked to predict the performance of students whom they did not meet, based solely on the students’ course schedules and past GPAs. The results showed that the GPA predictions were significantly more accurate for students who were not interviewed, and that the interviews had been counterproductive. As part of the study design beyond interview versus no interview, the researchers instructed interviewees to respond randomly to interview questions in some of their interviews. In half of the interviews, interviewees were instructed to answer honestly, and in the other half were instructed to answer randomly using a specific formula, responding to a group of yes/no or this/that questions. It was fascinating to note that not a single interviewer noticed that he/she was conducting an interview with random answers; and all the students who unwittingly conducted random interviews rated the degree to which they got to know the interviewee slightly higher on average than those who conducted honest interviews. Dana observed: “the key psychological insight here is that people have no trouble turning any information into a coherent narrative…. People can’t help seeing signals, even in noise.” ³

Dana also described an example of the lack of utility in interviews for distinguishing candidates from the University of Texas Medical School at Houston from 1979. The medical school was ordered by the Legislature to increase its incoming class size by 50 students late in the student selection season. The additional students who were accepted to fill those 50 seats had reached the interview phase, but following their interviews they had been rejected. Later research showed no difference in terms of academic performance, clinical performance (including human interaction), honors earned and attrition between those 50 students and the rest of the class who had had successful interviews leading to admission. ⁴ But this isn’t the only example: a 1991 study from the Brown University Program in Medicine examining medical school and residency performances of students admitted with or without an admission interview showed no differences in outcomes or academic performance. ⁵ This would suggest that interviews have no benefit in predicting whether some students are going to perform better than others! At first, it may seem hard to believe that such an established process as interviewing provides no useful information, but this reluctance is a bias that Dana studied specifically as well.

In a fascinating sequel to the first study about GPA prediction, the researchers explained what they had done, including their findings, to another group of student subjects. Thereafter, they asked their subjects what information they would like to have in order to make a GPA prediction. The subjects all still wanted interviews, believing that, despite the data, they would be able to glean useful information from personal interviews on which to base their predictions. Dana’s subjects are not alone in their belief. In fact, there have been numerous studies since the 1980s and earlier that have pointed to biases in hiring that are caused by unstructured interviewing practices. Yet, most of us still rely heavily on unstructured interviews for admissions and hiring.

Although most of us think that we can learn something useful from personal interviews, usually unstructured, Dana believes that we are wrong. He recognizes that unstructured interviews will continue but advises that we be very conservative in the interpretations and conclusions we draw from them. He also counsels that structured interviews, with all candidates receiving the same questions, provide better information, as do interviews directed specifically to test job-related skills. There are data to support this approach, from the field of medical admissions; some US medical schools have incorporated structured interviews such as the multiple-mini interview (MMI) into their interview procedure, and some have replaced traditional interviews entirely. In a 2017 study of 5 California medical schools, where prospective students often undergo multiple interviews at multiple comparable institutions, both within school and between school inter-rater reliability were greater for MMIs compared to traditional (unstructured) interviewing scores. ⁶ Follow-up data from these same schools, published in 2019, showed that MMI scores were more strongly correlated with subsequent academic performance in medical school than traditional interview scores. ⁷ While this research makes conclusions only about the consistency of outcomes comparing structured and unstructured interviews, the findings may offer reasons to consider the merits of structuring interviews.

Like Dana, we believe that the unstructured interview will remain at least partially impervious to the literature’s consistent criticisms and will continue However, in the past few years, the author (RTS) has started incorporating structure into evaluative interviews while leading search committees, and they have proven as useful as the literature suggests. Whether we are interviewing potential secretaries, students, associates, hospital administrators, political candidates, or others, most of us have considerable faith in our abilities to “read people.” Perhaps we are not as good at doing so as we think. Clearly, the subject requires further research and maybe an extra dose of humility and circumspection.