A software metric suite for evaluating the usability of Learning Management Systems

Abstract

In the last decade the World Wide Web has grown rapidly in both acceptance and application across a wide range of business, scientific, educational and societal contexts. From the mid 1990's there has been a move from static Internet sites to dynamic database driven Internet applications, Learning Management Systems (LMS) being an example of the latter in the education domain. Although intrinsically different from previous information systems because of the ubiquitous character of the Internet, many issues associated with more traditional software engineering projects are observable in the dynamics of newer Internet-based systems development activities. Usability is one of the most significant of those issues carried over into the contemporary Internet systems development environment that are crucial to their acceptance. Deploying a system that is not usable can greatly increase post-implementation maintenance and can incur training costs that could have been avoided if more rigorous usability testing had been conducted during development.

Numerous usability testing methods exist. While often very effective, most require input from test subjects and can therefore be costly in time and effort. A further drawback to these methods is that they are generally applied close to the time of final deployment, meaning that the developers do not receive sufficiently detailed feedback to enable them to focus on fixing the offending elements that caused the usability problem in the first place. This suggests that developers need a low cost usability assessment method that can be used iteratively throughout the development process, giving them greater insight into usability problems than current methods allow.

The research described in this thesis is primarily concerned with establishing and evaluating a cost-effective metric suite for determining the usability of Learning Management Systems. These LMS are complex interactive Internet applications that are increasingly integral to IT assisted teaching and learning environments.

The research issues raised in this thesis lead to the development of the LMS Metric Suite. This metric suite is based upon a number of design and contextual criteria (a combination of intuitive characteristics and criteria established in the research field of human-computer interaction), enabling developers to evaluate the usability of LMS without requiring opinionative, subjective and expensive input from users. By reviewing the outcome of the calculated metric suite the developer should be able to clearly identify weaknesses within the usability of the design.

A study is conducted that compares two common usability methods and the LMS Metric Suite against two versions of a deployed LMS in order to evaluate the effectiveness of the metric suite in establishing (approximating) usability acceptance potential.

The results indicate that the LMS Metric Suite is a useful tool for developers to gauge the usability of their LMS and provides them with sufficiently detailed feedback to allow them to fix the offending elements. As such it is an effective complementary method that should be used in conjunction with later-phase user-based testing.