Identifying interview selection tools used in medical school admissions interviews that are reliable and valid is critically important as almost all applicants chosen to interview are already highly competitive. Multiple Mini Interviews (MMI’s) are one tool that has been proven to be reliable and valid compared to other methods of selecting candidates including personal interviews, reference letters, personality testing, emotional intelligence, and situational judgment tests. MMI’s have been found to accurately measure non-cognitive traits including interpersonal skills, communication skills, and professionalism as well as helping to control for context specificity. The continued use of MMI’s is supported due to their reliability and validity as an effective tool in the medical school admissions process. Developing effective, reliable, and valid, as well as acceptable tools in the selection of medical school students has proven quite challenging. One of the most significant problems is that medical school applicants chosen to attend an interview day are anything but a random sample from the population; they are all highly competitive.¹ A method, growing in popularity, to help differentiate such similar candidates is the multiple mini interview (MMI). Not only has the MMI been proven reliable and valid^2‒6 it has also taken into account other factors influencing candidate selection such as non-cognitive characteristics, context specificity, and has been found appealing by both interviewers and applicants. According to¹ the Medical College Admissions Test (MCAT), and the MMI have been found to show predictive validity while other more traditional approaches such as the personal interview, reference letters, personality testing, emotional intelligence, and situational judgment tests are not as stringent.

McMaster University developed the MMI as a “multiple sample approach to the personal interview”⁷ to help capture non-cognitive traits such as interpersonal skills and professionalism which are extremely difficult to assess with traditional one-on-one interviews. One reason the authors developed the MMI was to help reduce the problems associated with context specificity. The study at McMaster University found the MMI to be reliable as well as support that variance due to candidate-station interaction was great than due to variance due specifically to the candidate thus supporting the idea that the MMI could help reduce the problems associated with context specificity compared to the individual interview. Again, both candidates and interviewers found the process of the MMI acceptable. Reliability and validity studies are extremely critical because interviewers and medical schools may not be able to observe the benefits of the MMI over the individual interview. Subsequent to the work at McMaster University, reliability and validity has been repeatedly illustrated.^2‒6 In 2009 and 2010 Hamburg University conducted a pilot study using MMI’s for medical school admission and reviewed cost cutting measures.³ To help increase reliability, the investigators discovered that it was more beneficial to increase the number of MMI stations as opposed to the number of raters and that it was also helpful to invest in rater training, scenario development, and scoring procedures. Interviewer characteristics such as skill, bias, leniency or stringency, and context specificity can all make assessing reliability, validity, and acceptability of the multiple mini-interview quite challenging.² However, studies have found that MMI’s were more effective than traditional interviews, and, in fact, both applicants and interviewers typically prefer the MMI format over typical interview formats.² In yet another study⁴ the reliability and validity of measuring non-cognitive characteristics using MMI’s found that 28% of variance between participant scores was due to interviewer characteristic such as subjectivity and 40% of the variance in scores was due differences in views the interviewers had for particular candidates. Interestingly, less than 2% of participants in this study would have failed the MMI. The fact that most applicants pass MMI’s is a significant problem because it limits the effective use of MMI’s even if the questions are reliable and valid. Developing MMI’s that are reliable, valid, and differentiate applicants would be ideal. Often the MMI will use situationally based questions as opposed to past-behaviouralbehavioral questions⁵ often used in traditional individual interviews. In the study by Yoshimura et al., it was found that MMI’s were reliable utilizing seven stations with one examiner per station and that all participants perceived MMI’s as more fair than traditional interview methods. Interestingly, applicants liked the situationally based questions better while the interviewers preferred the past-behaviouralbehavioral questions; however, both types of questions were found equally reliable. More studies are needed to help improve reliability and predictive validity. One extremely important concept in effective MMI’s is validating the question bank.⁸ The study by Roberts, et. al utilized item response theory (IRT) to validate the question bank and results discovered that MMI’s may be measuring cognitive reasoning skills in addition to non-cognitive skills. Validating the question bank appears to be what is most critical for the effective use of MMI’s. Also, the possibility the MMI’s may also be measuring additional cognitive reasoning skills is intriguing and a great area for future research as the MMI could possibly be assessing more than it was originally intended to. The study also focused on the importance of validity and fairness to help ensure that candidates were not negatively impacted due to the diverse background candidates came from.

As stated above, the MMI has been found to be reliable and valid meaning that the MMI consistently measures what it was designed to measure which are medical school applicants’ non-cognitive traits including interpersonal skills, communication skills, and professionalism.^2‒11 In fact, the MMI has been found to assess these non-cognitive traits better than traditional interviews.² One concern with assessing non-cognitive traits is that they may not occur exclusively of cognitive characteristics.¹² This may mean that it is quite difficult to completely separate cognitive and non-cognitive traits using interviews; the process is likely not a binary relationship, but rather, a holistic relationship that is much more complex than can be fully captured during an interview process. Evidence has been found to support that the MMI is a valid tool in assessing the non-cognitive characteristics of interpersonal and communication skills prior to entering residency training program.⁶ The Reading Health System interviewed six hundred nineteen candidates between 2011 and 2014 and ultimately selected 52 residents citing that the MMI is a valid tool for assessing the non-cognitive characteristics of interpersonal and communication skills. The MMI has been found to have more predictive power than traditional interviews and requires fewer person-hours to conduct ¹³. The authors did acknowledge that the MMI does require more effort in preparation and number of required rooms, however, they also acknowledged that the interview process in general is necessary as a recruitment opportunity as well as showing what their program has to offer.

One reason the MMI was developed was to help reduce the problems associated with context specificity compared to.⁷ Eva et al.,⁷ found that the MMI could help reduce the problems associated with context specificity compared to the individual interview. Again, both candidates and interviewers found the process of the MMI acceptable.

The reliability and validity of MMI’s has been repeatedly shown^2‒22 and supports the continued use of the MMI in medical school admissions. Validating the question bank seems to be of extreme importance in the effective use of the MMI as well as using and/or increasing interview stations as opposed to only increasing the number of interviews. With the abundance of reliability and validity data, as well as preference of the MMI over individual interviews by both applicants and interviewers, it would seem to lack foresight to prematurely discontinue the use of the MMI at this point.

None.

Author declares there are no conflicts of interest.

None.

1. Siu E, Reiter HI. Overview: what’s worked and what hasn't as a guide towards predictive admissions tool development. Adv Health Sci Educ Theory Pract. 2009;14(5):759‒775.
2. Eva KW, Reiter HI, Rosenfeld J, et al. The ability of the multiple mini-interview to predict preclerkship performance in medical school. Acad Med. 2004;79(10 Suppl):S40‒S42.
3. Eva KW, Reiter HI, Rosenfeld J, et al. The relationship between interviewers’ characteristics and ratings assigned during a multiple mini-interview. Acad Med. 2004;79(6):602‒629.
4. Hissbach JC, Sehner S, Harendza S, et al. Cutting costs of multiple mini-interviews-Changes in reliability and efficiency of the Hamburg medical school admission test between two applications. BMC Med Educ. 2014;14:54.
5. Oluwasanjo A, Wasser T, Alweis R. Correlation between MMI performance and OSCE performance-a pilot study. J community Hosp Intern Med Perspect. 2015;5(3):27808.
6. Roberts C, Clark T, Burgess A, et al. The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training. BMC Med Educ . 2014;14:169.
7. Eva KW, Rosenfeld J, Reiter HI, et al. An admissions OSCE: The multiple mini-interview. Med Educ. 2004;38(3):314‒326.
8. Roberts C, Zoanetti N, Rothnie I. Validating a multiple mini-interview question bank assessing entry-level reasoning skills in candidates for graduate-entry medicine and dentistry programmes. Med Edu. 2015;43(4):350‒359.
9. Fraga JD, Oluwasanjo A, Wasser T, et al. Reliability and acceptability of a five-station multiple mini-interview model for residency program recruitment. J community Hosp Intern Med Perspect. 2013;3(3‒4):10.3402.
10. Hissbach JC, Sehner S, Harendza S, et al. Cutting costs of multiple mini-interviews- changes in reliability and efficiency of the Hamburg medical school admission test between two applications. BMC Med Educ. 2015;14:54.
11. oberts C, Clark T, Burgess A, et al. The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training. BMC Med Educ. 2014;14:169.
12. Yoshimura H, Kitazono H, Fujitani S, et al. Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: A reliability and acceptability comparison. BMC Med Educ. 2015;15:75.
13. Lemay JF, Lockyer JM, Collin VT, et al. Assessment of non-cognitive traits through the admissions multiple mini-interview. Med Educ. 2007;41(6):573‒579.
14. Eva KW, Reiter HI, Trinh K, et al. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ. 2009;43(8):767‒775.
15. Rosenfeld JM, Reiter HI, Trinh K, et al. A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews. Adv Health Sci Educ Theory Pract. 2008;13(1):43‒58.
16. Burkhardt JC, Stansfield RB, Vohra T, et al. Prognostic Value of the Multiple Mini-Interview for Emergency Medicine Residency Performance. J Emerg Med . 2015;49(2):196‒202.
17. Campagna Vaillancourt M, Manoukian J, Razack S, et al. Acceptability and reliability of multiple mini interviews for admission to otolaryngology residency. Laryngoscope. 2014;124(1):91‒96.
18. Dore KL, Hanson M, Reiter HI, et al. Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool. Acad Med . 2006;81(10 Suppl):S70‒S73.
19. Dore KL, Kreuger S, Ladhani M, et al. The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions. Acad Med. 2010;85(10 Suppl):S60‒S63.
20. Finlayson HC, Townson AF. Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview. Am J Phys Med Rehabil . 2011;90(4):330‒335.
21. Husbands A, Dowell J. Predictive validity of the Dundee multiple mini-interview. Med Educ. 2013; 47(7):717‒725.
22. Yoshimura H, Kitazono H, Fujitani S, et al. Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: a reliability and acceptability comparison. BMC Med Educ . 2015;15:75.