Abstract
Diagnostic and therapeutic decision-making is perhaps the most important role of doctors as healthcare professionals. The majority of these judgements by doctors are correct, but when erroneous decisions are made it can lead to adverse events with significant implications for patient welfare. Doctors require not only sufficient knowledge and skill to make decisions, but also the capacity to know when they have reached the limit of that knowledge and skill. This capacity is self-monitoring, which is the activity of reflection-in-action as part of decision-making in daily practice. As it is a critical component of practice, medical education needs to assist students to develop self-monitoring.
One approach to developing self-monitoring is to embed it into medical education assessment. The main purposes of assessment include guiding and motivating learning and informing decisions on whether students should progress to subsequent levels. Research into assessment related to self-monitoring is relatively sparse. Belief about one’s ability to undertake actions and about the consequences of those actions are at the core of the theories that underpin self-monitoring. Assessment of self-monitoring could be informed by evaluating certainty in assessment item responses and safety of item responses.
There is a need to develop an assessment technique for self-monitoring that would include items based on authentic clinical practice and that would incorporate both certainty in, and safety of, item responses. Certainty could be assessed by having students rate their certainty in their multiple-choice examination responses. Safety could be assessed by having experts rate the potential outcome of each option in a multiple-choice item.
This thesis includes four separate studies. Across these studies medical students sat a variety of multiple-choice tests which were of different stakes, length and content. They were designed to introduce and evaluate the inclusion of certainty in, and safety of, responses. The studies included a variety of outcomes with evaluation at cohort and individual student level for: feasibility; scoring systems; progression decision-making; correctness for levels of certainty; safeness for levels of certainty; and the influence of student ability, experience, gender, ethnicity, and feedback on correctness-in and safeness-of responses for levels of certainty.
The main findings were that:
1) It was feasible to incorporate certainty in responses, and to a lesser extent, safeness of responses.
2) The reliability of responses was greatest for certainty, less for correctness and least for safeness.
3) Students of all levels of ability and experience demonstrated increasing correctness with increasing certainty.
4) Unsafe responses occurred across all levels of ability, experience and certainty.
Correctness, correctness for levels of certainty, and safeness for levels of certainty have been demonstrated to vary in different ways, suggesting that they measure different constructs and therefore could be important to introduce to medical student education programmes. Multiple-choice question tests with certainty-in and safeness-of responses introduced early in medical education and training is a potential means to better understand and enhance learning of self-monitoring of clinical decisions.