Abstract
Phenomenon: Programmatic assessment and competency-based education have highlighted the need to make robust high-stakes assessment decisions on learner performance from evidence of varying types and quality. Without guidance, lengthy deliberations by decision makers and competence committees can end inconclusively with unresolved concerns. These decisional dilemmas are heightened by their potential impacts. For learners, erroneous decisions may lead to an unjustified exit from a long-desired career, or premature promotion to clinical responsibilities. For educators, there is the risk of wrongful decision-making, leading to successful appeals and mistrust. For communities, ill-prepared graduates risk the quality and safety of care. Approaches such as psychometric analyses are limited when decision-makers are faced with seemingly contradictory qualitative and quantitative evidence about the same individual. Expertise in using such evidence to make fair and defensible decisions is well established in judicial practice but is yet to be practically applied to assessment decision-making. Approach: Through interdisciplinary exchange, we investigated medical education and judicial perspectives on decision-making to explore whether principles of decision-making in law could be applied to educational assessment decision-making. Using Dialogic Inquiry, an iterative process of scholarly and mutual critique, we contrasted assessment decision making in medical education with judicial practice to identify key principles in judicial decision-making relevant to educational assessment decisions. We developed vignettes about common but problematic high-stakes decision-making scenarios to test how these principles could apply. Findings: Over 14 sessions, we identified, described, and applied four principles for fair, reasonable, and transparent assessment decision-making. These were:
The person whose interests are affected has a right to know the case against them, and to be heard.
Reasons for the decision should be given.
Rules should be transparent and consistently applied.
Like cases should be treated alike and unlike cases treated differently.
Reflecting our dialogic process, we report findings by separately presenting the medical educator and judicial perspectives, followed by a synthesis describing a preferred approach to decision-making in three vignettes. Insights: Judicial principles remind educators to consider both sides of arguments, to be consistent, and to demonstrate transparency when making assessment decisions. Dialogic Inquiry is a useful approach for generating interdisciplinary insights on challenges in medical education by critiquing difference (e.g., the meaning of objectivity) and achieving synthesis where possible (e.g., fairness is not equal treatment of all cases). Our principles and exemplars provide groundwork for promoting good practice and furthering assessment research toward fairer and more robust decisions that will assist learning.