The efficacy of medical student selection tools in Australia and New Zealand

Objectives: To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools.

Design, setting and participants: Data from four consecutive cohorts of students (3378 students, enrolled 2007–2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt.

Main outcome measures: Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes.

Results: For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44–1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00–0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others.

Conclusions: An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A “sufficient evidence” selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.