Think-aloud protocol and heuristic evaluation of non-immersive, desktop photo-realistic virtual environments

Abstract

The concept of Virtual Reality (VR), a three-dimensional, computer-generated environment that allows for a single or multiple users to interact, navigate, respond, and experience a synthesized world modelled from the real world, has provided social, scientific, economic and technological change since its inception in the early 1960's. Since that time, VR has also evolved into many forms and taken different tangents. One form that it has taken is non-immersive desktop photo-realistic Virtual Environments (VEs), where real-world still images are joined together to create 360 degree panoramas of places and objects in time. The concept has become increasingly popular since its creation using Apple QuickTime Virtual Reality Authoring Studio (QTVRAS) in 1995.

This study is about the evaluation of non-immersive desktop web-based photo-realistic virtual environments using a previously applied evaluation method for non-immersive desktop 3D multi-user environments conducted by Koykka, Ollikainen, Ranta-aho, Milszus, Wasserroth and Friedrich (1999). The two qualitative evaluation methods are think-aloud protocol analysis and heuristic evaluation. There are three aims in this study: (1 ) to determine whether applying the same usability evaluation methodologies result in additional usability guideline categories: 3D environments should provide support for orientation, navigation and movement, real world metaphors need to be clearly understandable and avoidance of delay and waiting periods in performance (identified from the Koykka et at. study); (2) to determine whether think-aloud protocol or usability heuristic evaluation is a better evaluation method, for identifying usability problems in desktop, photo-realistic virtual environments; and (3) to determine whether large-scale desktop, photo-realistic virtual environments will have more usability problems than small-scale non-immersive, desktop, photo-realistic virtual environments.

It is hoped that by investigating these aims that design guidelines for desktop, photo-realistic virtual environments may be derived.

Twelve subjects took part in evaluating two different web-based photo-realistic VEs using the Think-Aloud Protocol (TAP) and Jakob Nielsen's usability Heuristic Evaluation (HE). The think-aloud protocols were videotaped and the heuristic evaluation results were word-processed. A variation of the grounded theory research method called Emergent Themes Analysis (ETA) was used to "distil" the recorded narratives into broad themes.

The results show that ETA derived four broad themes: functionality, interaction, appearance and user comments. Within each broad theme, sub-themes were also derived to produce possible design guidelines for desktop, photo-realistic virtual environments. The results also show that the functionality and interaction broad themes were similar to the categories suggested by Koykka et al. (1999). Furthermore, the results indicate that small-scale non-immersive, desktop, photo-realistic virtual environments had more usability problems than large-scale non-immersive, desktop, photo-realistic virtual environments, contrary to the hypothesis that large-scale VEs will have more usability problems due to its complexity.

Using the ETA method to analyze the data, the think-aloud protocol derived more themes compared to the heuristic evaluation – suggesting that TAP is a better usability evaluation method than HE in this type of study. However, a combination of the two qualitative methods has identified a greater number of usability problems, supporting the need for triangulation of research methods.

It is hoped that these new design guidelines will allow for more usable design of non-immersive desktop, photo-realistic virtual environments. Furthermore, it is hoped that this study provides a beginning of future developments of usability evaluation methods for non-immersive desktop, photo-realistic virtual environments. Regardless of its future use, usability evaluations of any virtual environments are still needed.