Abstract
In large undergraduate classes, it is time-consuming, costly and seldom practical for the teacher to provide individualised feedback to students on their written responses to questions. A request for advice in relation to these issues, from the coordinator of a large first-year health sciences course at the University of Otago, motivated the research described in this thesis.
An intelligent tutoring system (ITS) engages students in a dialogue; students enter their contributions to the dialogue as free text and the system gives a response. Such systems, which employ natural language as their interface, are called dialogue-based ITS or tutorial dialogue systems; they offer some promise for supporting and enhancing student understanding of key concepts through the provision of individualised feedback. The appeal of tutorial dialogue is that questions are embedded in a tutorial plan: the questions arise in a meaningful context and concepts and ideas are linked together in a coherent form. This allows each student contribution to be individually assessed.
However, ITS are not currently in widespread use in higher education settings and there has been considerable controversy around their application in this context. Practical issues relating to the time and cost for development, the difficulty of adapting to specific teaching contexts, and pedagogical objections which relate to the idea of student modelling are among some of the barriers to their widespread adoption. In this thesis, a rationale is presented for revisiting tutorial dialogue systems in the context of large-class teaching.
Two broad goals for this research are delineated. The first goal was to design, build and evaluate a new tutorial dialogue system for the cardiovascular section of a first-year undergraduate health-sciences paper. The new system is firmly empirically-based, with both the teaching context and real student responses to questions integral to its design and implementation.
The second goal was to determine whether a tutorial dialogue system, which provides students with the opportunity to practise writing answers to short-answer questions and gives automated feedback about these answers, would result in improved student performance. In order to explore the second goal, two versions of the new system were developed. The first version of the system required students to type their response to questions as free-text; in the second version, students selected the answer they preferred from a menu of options. Student volunteers were divided into three groups: a control group, a free text tutorial group and a menu-based tutorial group. The performance between the two tutorial groups and the control group were compared to test whether free-text entry conferred any performance advantage over selection from a menu of options.
The design and implementation of the new tutorial dialogue system is described in detail and some limitations are discussed. The evaluation of the new dialogue system with 578 student volunteers in a real-class setting is described. Student perceptions of the system were broadly positive and there was strong uptake of the system compared with an earlier prototype. The experiment, which was set up specifically to test the performance of the system overall, as well as to establish if there are differences between free-text and menu-based versions, found student performance gains did occur among students who used either version of the new tutorial dialogue system but no differences were found between the two versions.
The main conclusion to be drawn from this research is that the new system can be deployed in a large-class setting and, at least in the context of first-year health sciences undergraduate courses, is likely to find acceptance with students, in addition to having a positive impact on their performance. The development of a stable platform for the further study of tutorial dialogues and the automated creation of a large corpus of tutorial dialogues are spin-off benefits from the research. Finally, this research is a small contribution towards getting contemporary tutorial dialogue systems back on the educational agenda.