University of Tasmania
Browse

File(s) under permanent embargo

Inter-rater reliability: comparison of checklist and global scoring for OSCEs

journal contribution
posted on 2023-05-17, 14:20 authored by Malau-Aduli, BS, Mulcahy, S, Emma Warnecke, Petr OtahalPetr Otahal, Teague, PA, Richard TurnerRichard Turner, Van der Vleuten, C
Objective Structured Clinical Examinations (OSCEs) have been used globally in evaluating clinical com- petence in the education of health professionals. Despite the objective intent of OSCEs, scoring methods used by examiners have been a potential source of measurement error affecting the precision with which test scores are determined. In this study, we investigated the differences in the inter-rater reliabilities of objective checklist and subjective global rating scores of examiners (who were exposed to an online training program to standardise scoring techniques) across two medical schools. Examiners’ perceptions of the e-scoring program were also investigated. Two Australian universities shared three OSCE stations in their end-of-year undergraduate medical OSCEs. The scenarios were video-taped and used for on-line examiner training prior to actual exams. Examiner ratings of performance at both sites were analysed using generalisability theory. A single facet, all random persons by raters design [PxR] was used to measure inter-rater reliability for each station, separate for checklist scores and global ratings. The resulting variance components were pooled across stations and examination sites. Decision studies were used to meas- ure reliability estimates. There was no significant mean score difference between examination sites. Variation in examinee ability accounted for 68.3% of the total variance in checklist scores and 90.2% in global ratings. Rater contribution was 1.4% & 0% of the total variance in checklist score and global rating respectively, reflecting high inter-rater reliability of the scores provided by co-examiners across the two schools. Score variance due to interaction and residual error was larger for checklist scores (30.3% vs 9.7%) than for global ratings. Reproducibility coefficients for global ratings were higher than for checklist scores. Survey results showed that the e-scoring package facilitated consensus on scoring techniques. This approach to examiner training also allowed examiners to calibrate the OSCEs in their own time. This study revealed that inter-rater reliability was higher for global ratings than for checklist scores, thus providing further evidence for the reliability of subjective examiner ratings.

History

Publication title

Creative Education

Issue

6A

Pagination

937-942

ISSN

2151-4755

Department/School

Tasmanian School of Medicine

Publisher

Scientific Research Publishing, Inc.

Place of publication

5005 Paseo Segovia, Irvine, CA 92603, USA

Rights statement

Copyright 2012 SciRes

Repository Status

  • Restricted

Socio-economic Objectives

Assessment, development and evaluation of curriculum

Usage metrics

    University Of Tasmania

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC