Validity of a Virtual Reality-Based Clinical Case for Assessment of Clinical Competence

Abstract

Reliable and valid assessment of the clinical competence of medical students and doctors is essential for the safety of patients. Current modes of assessment are limited in their ability to evaluate some key aspects of competence, such as clinical reasoning ability and timely decision-making. The aims of this study were to assess the validity of a virtual reality-based clinical case as a method of assessment of clinical competence. In addition, this study intended to specifically examine the capacity of the virtual reality case format to assess clinical reasoning ability. The Otago Virtual Hospital is a virtual reality-based computer programme in which the performance of doctors and students can be assessed while managing a simulated clinical case in real time. As a pilot study, 12 participants, (consisting of three cohorts comprised of third-year medical students, fifth-year medical students and qualified doctors) from Otago Medical School participated in a simulated clinical case. Their performance was measured by scoring their achievement of set outcomes representing optimal management; these were developed from expert opinion. Qualitative thematic analysis of case transcripts was undertaken to compare clinical reasoning skills.

Scores of performance from the virtual reality case showed qualified doctors achieved the highest scores, significantly higher than the third-year student group. Qualified doctors were also significantly better able to make correct full diagnoses and achieve safe clinical management compared with the student cohorts. These results showed some significant differences between groups at different stages of medical training, hence supporting the construct validity of the virtual-reality based clinical case. Thematic analysis to identify clinical reasoning themes indicated that compared to the student cohorts, the qualified doctor group was better able to transform information into key clinical concepts, generate more accurate diagnoses, and generate correct diagnoses more efficiently. With increasing clinical experience there was a superior ability to communicate clinical information succinctly and precisely, and to construct effective patient management plans.

The virtual reality based clinical case provided an authentic clinical task. Within the programme, overall performance and clinical reasoning abilities could be assessed by analysis of a summary admission note at the conclusion of the case. These results suggest that simulated virtual cases may provide a valid and rapid means of assessing clinical competence, and can provide more comprehensive information about clinical reasoning ability than traditional means of assessment.