Abstract
Commentators have argued that achievement measures based on teachers’ judgments lack validity (Ward & Thomas, 2016) and introduce bias (Meissel et al., 2017). This thesis addresses this issue by exploring whether decision frameworks can enhance the validity of teachers’ summative judgments. A decision framework is a systematic process used to support and guide judgments. The thesis focusses on a framework developed in New Zealand called the Progress and Consistency Tool (PaCT). Intended for use with students in Years 1 to 10, the PaCT supports teachers to judge how well their students are achieving in reading, writing, and mathematics.
The thesis presents three studies designed to address different aspects of validity. The first study investigated the quality of the measures produced by a decision framework. It used multilevel modelling to explore the relationship between PaCT scores and scores on standardised tests. Study 2 investigated the practical viability of a decision framework. Using survey data collected from teachers and principals in schools that had committed to use the PaCT, the study compared the reactions of users who had some experience of the PaCT with those who were about to use it for the first time. The final study explored the relationship between the content of a decision framework and the measures the framework produces. The study involved the development of a methodology that could be used to describe each of the Rasch measurement scales that underpin the PaCT.
Study 1 found a strong linear relationship between the scores generated by teachers using the PaCT and scores on standardised tests. This supported a hypothesis that both approaches were measuring the same construct. Although the series of multilevel modelling analyses did not find consistent evidence of systematic bias in teachers’ PaCT judgments, the results did suggest a level of idiosyncrasy in the way that teachers made use of the PaCT.
Study 2 showed that educators who had experience using the PaCT were generally more positive about it than those who had no experience. However, school role mattered. On average, classroom teachers recorded lower Attitude and Practicality scores regarding the PaCT than educators who had school leadership roles.
The findings from Study 3 showed that it was possible to develop a robust methodology, based on the Rasch model, to link descriptions of the rich illustrations that form the content of the PaCT to most-probable locations on the underpinning measurement scales. The approach was applied by subject experts to successfully create a rich description of each of the PaCT scales.
Taken together, the findings from the three studies suggest that decision frameworks can enhance the validity of teachers’ summative judgments. However, concerns around idiosyncrasy and practicality did emerge. It is argued that these issues could be mitigated by incorporating decision frameworks into what Wilson described as a “community of judgment” (2005a). Such a community would promote shared responsibility for developing appropriate understandings of the frameworks, including how they are best deployed in diverse educational settings.