Show simple item record

dc.contributor.authorSallis, Adrian Jamesen_NZ
dc.date.available2011-04-07T03:12:35Z
dc.date.copyright2005-10-14en_NZ
dc.identifier.citationSallis, A. J. (2005, October 14). A software metric suite for evaluating the usability of Learning Management Systems (Thesis, Master of Arts). Retrieved from http://hdl.handle.net/10523/1282en
dc.identifier.urihttp://hdl.handle.net/10523/1282
dc.description.abstractIn the last decade the World Wide Web has grown rapidly in both acceptance and application across a wide range of business, scientific, educational and societal contexts. From the mid 1990's there has been a move from static Internet sites to dynamic database driven Internet applications, Learning Management Systems (LMS) being an example of the latter in the education domain. Although intrinsically different from previous information systems because of the ubiquitous character of the Internet, many issues associated with more traditional software engineering projects are observable in the dynamics of newer Internet-based systems development activities. Usability is one of the most significant of those issues carried over into the contemporary Internet systems development environment that are crucial to their acceptance. Deploying a system that is not usable can greatly increase post-implementation maintenance and can incur training costs that could have been avoided if more rigorous usability testing had been conducted during development. Numerous usability testing methods exist. While often very effective, most require input from test subjects and can therefore be costly in time and effort. A further drawback to these methods is that they are generally applied close to the time of final deployment, meaning that the developers do not receive sufficiently detailed feedback to enable them to focus on fixing the offending elements that caused the usability problem in the first place. This suggests that developers need a low cost usability assessment method that can be used iteratively throughout the development process, giving them greater insight into usability problems than current methods allow. The research described in this thesis is primarily concerned with establishing and evaluating a cost-effective metric suite for determining the usability of Learning Management Systems. These LMS are complex interactive Internet applications that are increasingly integral to IT assisted teaching and learning environments. The research issues raised in this thesis lead to the development of the LMS Metric Suite. This metric suite is based upon a number of design and contextual criteria (a combination of intuitive characteristics and criteria established in the research field of human-computer interaction), enabling developers to evaluate the usability of LMS without requiring opinionative, subjective and expensive input from users. By reviewing the outcome of the calculated metric suite the developer should be able to clearly identify weaknesses within the usability of the design. A study is conducted that compares two common usability methods and the LMS Metric Suite against two versions of a deployed LMS in order to evaluate the effectiveness of the metric suite in establishing (approximating) usability acceptance potential. The results indicate that the LMS Metric Suite is a useful tool for developers to gauge the usability of their LMS and provides them with sufficiently detailed feedback to allow them to fix the offending elements. As such it is an effective complementary method that should be used in conjunction with later-phase user-based testing.en_NZ
dc.subjectUsabilityen_NZ
dc.subjectInterneten_NZ
dc.subjectcost-effectiveen_NZ
dc.subjectmetric suiteen_NZ
dc.subjectLearning Management Systems,en_NZ
dc.subject.lcshQ Science (General)en_NZ
dc.titleA software metric suite for evaluating the usability of Learning Management Systemsen_NZ
dc.typeThesisen_NZ
dc.description.versionUnpublisheden_NZ
otago.date.accession2006-08-26en_NZ
otago.schoolInformation Scienceen_NZ
thesis.degree.disciplineInformation Scienceen_NZ
thesis.degree.nameMaster of Arts
thesis.degree.grantorUniversity of Otagoen_NZ
thesis.degree.levelMasters Thesesen_NZ
otago.interloanyesen_NZ
otago.openaccessAbstract Only
dc.identifier.eprints336en_NZ
otago.school.eprintsInformation Scienceen_NZ
dc.description.referencesAdrion, W. R., Barnstad, M. and Cherniaysky, J. (1982). Validation, verification, and testing computer software, ACM Computing Surveys 14(2): 159-192. AL-Qaimari, G. and McRostie, D. (1999). KALDI: A computer-aided usability engineering tool for supporting testing and analysis of humancomputer interaction., Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces, Kluwer Academic Publishers. Albrecht, A. J. (1985). Function points help manager assess applications, Computerworld (SR/20): 20-21. Anderson, C., Domingos, P. and Weld, D. (2001). Personalizing Web Sites for Mobile Users, Proceedings of the 10th International WWW Conference, ACM Press, pp. 565-575. Anderson, R., Homer, A., Sussman, D. and Blexrud, C. (1999). Professional Active Server Pages 3.0., Birmingham, Wrox Press Inc. Arroyo, I., Beck, J., Beal, C. R. and Woolf, B. (2003). Learning within the Zone of Proximal Development with the AnimalWatch intelligent tutoring system, Proceedings of the American Educational Research Association annual meeting. Accessed: 5/Mar/2004. ASU (2000). Arizona State University - 10 Critical Success Factors for Cyber Campuses, http://clte.asu.edu/active/Critical°420Succesa20Factors.doc. Accessed: 16/Feb/2002. Atkins, M. (1993). Evaluating interactive technologies for learning, Journal of Curriculum Studies 25(4): 333-342. Babiker, E. M., Fujihara, H. and Boyle, C. D. B. (1991). A metric for hypertext usability, Proceedings of the 9th annual international conference on Systems documentation, ACM Press, pp. 95-104. Bannon, L. (1991). Design at Work: Cooperative Design of Computer Systems. J. Greenbaum E M.Kyng(Eds.), Hillsdale, NJ: Lawrence Erlbaum Associates, chapter From human factors to human actors: The role of psychology and human-computer interaction studies in system design, pp. 25-44. Basili, V. and Perricone, B. T. (1984). Software Errors and Complexity: An Empirical Investigation, Communications of the ACM 27(1): 42-52. Beal, C. R., Arroyo, I., Royer, J. and Woolf, B. (2003). Wayang Outpost: An intelligent multimedia tutor for high stakes math achievement tests, Proceedings of the American Educational Research Association annual meeting. Accessed: 5/Mar/2004. Becker, R., McLaughlin, K. and Rebelsky, S. (1999). Project CLIO: Tools for tracking student use of course webs, World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA, Seattle, USA. Benbunan-Fich, R. (2002). Improving education and training with IT, Communications of the ACM 45(6): 94-99. Berkman, J. (1997). Web Design and Maintenance, 25th SIGUCCS Conference on User Services, Monterey, CA. Berners-Lee, T. (1990). WorldWideWeb: Proposal for a HyperText Project, http //www . w3 . org/Proposal . html. Accessed: 11/Jun/2003. Bevan, N. (1999). Quality in use: meeting user needs for quality, Journal of Systems and Software 49(1): 89-96. Bevan, N., Barnum, C., Cockton, G., Nielsen, J., Spool, J. and Wixon, D. (2003). The "magic number 5": is it enough for web testing?, CHI '03 extended abstracts on Human factors in computing systems, ACM Press, pp. 698-699. Bias, R. G. (1994). The pluralistic usability walkthrough: coordinated empathies, John Wiley and Sons, Inc., pp. 63-76. Biggs, J. B. and Telfer, R. (1987). The Process of Learning, 2 edn, Sydney, Prentice- Hall. Billsus, D., Brunk, C. A., Evans, C., Gladish, B. and Pazzani, M. (2002). Adaptive interfaces for ubiquitous web access, Communications of the ACM 45(5): 34-38. Blackboard (2004). Blackboard, http://www.blackboard. com/. Accessed: 5/Jan/2004. Blackmon, M. H., Poison, P. G., Kitajima, M. and Lewis, C. (2002). Cognitive walkthrough for the web, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 463-470. Bodart, F., Hennebert, A.-M., Leheureux, J.-M. and Vanderdonckt, J. (1994). Towards a Dynamic Strategy for Computer-Aided Visual Placement, 2nd Workshop on Advanced Visual Interfaces, Bari, Italy. Bodker, S., Ehn, P., Knudsen, J., Kyng, M. and Madsen, K. (1988). Computer support for cooperative design (invited paper), Proceedings of the conference on Computersupported cooperative work, ACM Press, pp. 377-394, Borges, J. A., Morales, I. and Rodrguez, N. J. (1996). Guidelines for Designing Usable World Wide Web Pages, Proceedings of the CHI '96 conference companion on Human factors in computing systems: common ground, ACM Press, pp. 277-278. http://doi.acm.org/10.1145/257089.257320. Botafogo, R. A., Rivlin, E. and Shneiderman, B. (1992). Structural analysis of hypertexts: identifying hierarchies and useful metrics, ACM Transactions on Information Systems (TOIS) 10(2): 142-180. Broder, A. and Kumar, R. (2000). Graph structure in the web, Proceedings of the 9th Intl. WWW Conference, ACM Press, pp. 309-320. Brown, A. and Palinscsar, A. (1989). In Lauren B. Resnick (Ed.) Knowledge, Learning and Instruction, Lawrence Erlbaum, Hillsdale, NJ, chapter Guided, Cooperative Learning and Individual Knowledge Acquisition. Brown, J. and Duguid, P. (1994). Borderline issues: Social and material aspects of design, Human-Computer Interaction 9(1): 3-36. Brusilovsky, P. and Maybury, M. T. (2002). From adaptive hypermedia to the adaptive web, Commun. ACM 45(5): 30-33. Brusilovsky, P., Ritter, S. and Schwarz, E. (1997). Distributed intelligent tutoring on the Web, 8th World Conference on Artificial Intelligence in Education, Amsterdam, IOS. Brusilovsky, P. and Rizzo, R. (2002). Map-based horizontal navigation in educational hypertext, Proceedings of the thirteenth ACM conference on Hypertext and hypermedia, ACM Press, pp. 1-10. Buie, E. (1999). HCI standards: a mixed blessing, Interactions 6(2): 36-42. Bygrave, J., Sallis, A., Matheny, J. and Carran, G. (1999). Collaborative Learning Online in International Business, Southeast Asian Institutions of Higher Learning (Asa,ihl), Auckland, New Zealand. Byrne, E. (2000). Evaluate Web Resources, http: //www. clubi o ie/webserch/ resources/content . htm. Accessed: 21/Dec/2003. C2T2 (2001). Online educational delivery applications: a web tool for comparative analysis, http://www. c2t2. ca/. Accessed: 12/Jan/2004. Canale, R. (1998). Review of Web based, Course Delivery Software Systems, http: //www.dls.au.com/papers/review-cds.html. Accessed: 1/May/2004. Center, 0. S. (2002). Tools, Reviews and Comparisons, http: //www.osc edu/ education/webed/Tools/review. shtml. Accessed: 8/Feb/2003. CERN (1998). The world wide web, CERN, http : //public . web . cern ch/public/ about/achievements/www/www.html. Accessed: 12/Jan/2004. CERN (2002). History and growth, CERN, http : //public . web . cern . ch/public/ about/achievements/www/history/history.html. Accessed: 15/Jan/2004. Chakrabarti, S., Gibson, D. and McCurley, K. (1999). Surfing the Web Backwards, Computer Networks 31: 1679-1693. Chi, E. H., Rosien, A., Supattanasiri, G., Williams, A., Royer, C., Chow, C., Robles, E., Dalal, B., Chen, J. and Cousins, S. (2003). The bloodhound project: automating discovery of web usability issues using the InfoScentπ simulator, Proceedings of the conference on Human factors in computing systems, ACM Press, pp. 505-512. Chi, M. T. H., Bassok, M., Lewis, M. W., Reinmann, P. and Galser, R. (1989). Self- Explanations: How Students Study and Use Examples in Learning to Solve Problems, Cognitive Science 13: 145-182. Choudhary, P. (1998). A Comparative Analysis Of Online Education Tools, http //www.visc . vt . edu/succeed/wwwframework/survey . html. Accessed: 14/Dec/2002. Cohen, E. (1994). Restructuring the classroom: Conditions for productive small groups, Review of Educational Research 64(1): 1-35. Coll, R. and Wingertsman, J. C. (1990). The effect of screen complexity on user preference and performance, International Journal of Human-Computer Interaction 2(3): 255-265. Collins, A. (1990). Educational Values and Cognitive Instruction, Lawrence Erlbaum, Hillsdale. NJ, chapter Cognitive Apprenticeship and Instructional Technology, In Idol, L. and Jones, B. F. (Eds.). Comber, T. and Maltby, J. (1996). Computer-aided design of user interfaces, J. Vanderdonckt. Namur, Belgium, Press Universitaries de Namur, chapter Investigating layout complexity, pp. 209-227. Conklin, J. (1987). Hypertext: An introduction and survey, IEEE Computer 20(9): 17- 41. Constantine, L. L. and Lockwood, L. A. D. (2002). Usage-centered engineering for web applications, IEEE Software 19(2): 42-50. http : //www. computer. org/software/ so2002/s2toc.htm. Conte, S. D., Dunsmore, H. E. and Shen, V. Y. (1986). Software Engineering Metrics and Models, Benjamin/Cummings Publishing Company, Inc., Menlo Park, California. Corry, M., Frick, T. W. and Hansen, L. (1997). User-centered design and usability testing of a web site: An illustrative case study, Educational Technology, Research and Development 45(4): 65-76. http : //education. indiana. edu/ -edfolks/persons/frick.html. Coulston, C. and Vitolo, T. M. (2001). A hypertext metric based on huffman coding, Proceedings of the twelfth ACM conference on Hypertext and Hypermedia, ACM Press, pp. 243-244. Crampes, M. and Ranwez, S. (2000). Ontology-supported and ontology-driven conceptual navigation on the world wide web, Proceedings of the eleventh ACM on Hypertext and hypermedia, ACM Press, pp. 191-199 Darken, R. P. and Sibert, J. (1996). Navigating in Large Virtual Worlds, International Journal of Human-Computer Interaction 8(1): 49-72. De Angeli, A., Matera, M., Costabile, M. F., Garzotto, F. and Paolini, P. (2000). Validating the sue inspection technique, Proceedings of the working conference on Advanced visual interfaces, ACM Press, pp. 143-150. DeBra, P., Houben, G., Kornatzky, Y. and Post, R. (1994). Information retrieval in distributed hypertexts, 4th RIAO-94 Conference, New York. DeMarco, T. (1982). Controlling Software Projects: Management, Measurement and Estimation, Yourdon Press, New York. Denaro, G. and Pezz, M. (2002). An empirical evaluation of fault-proneness models, Proceedings of the 24th international conference on Software engineering, ACM Press, pp. 241-251. der Aalst, W. V., Bra, P. D. Houben, G. J. and Kornatzky, Y. (1993). Computing Science Notes 93, chapter Browsing semantics in the "tower" model, p. 47. Desurvire, H. W. (1994). Faster, cheaper!! Are usability inspection methods as effective as empirical testing?, John Wiley and Sons, Inc., pp. 173-202. Desurvire, H. W., Kondziela, J. M. and Atwood, M. E. (1993). What is gained and lost when using evaluation methods other than empirical testing, Proceedings of the conference on People and computers VII, Cambridge University Press, pp. 89-102. Dicks, R. S. (2002). Mis-usability: on the uses and misuses of usability testing, Proceedings of the 20th annual international conference on Computer documentation, ACM Press, pp. 26-30. DiNucci, D., Giudice, M. and Stiles, L. (1998). Elements of web design, Technical report, Berkley CA, PeachPit Press. Dix, A., Finlay, J., Abowd, G. and Beale, R. (1993). Human-Computer Interaction, 2nd edn, London, Prentice Hall. Doubleday, A., Ryan, M., Springett, M. and Sutcliffe, A. (1997). A comparison of usability techniques for evaluating design, Proceedings of the conference on Designing interactive systems, ACM Press, pp. 101-110. Draper, S. (1998). Niche-based success in CAL, Computers and Education 30(1/2): 5– 8. Draper, S., Brown, M., Henderson, F. and McAteer, E. (1997). Integrative Evaluation: An Emerging Role for Classroom Studies of CAL, Computers in Education 26(1- 3): 17-32. Dugger, W. and Naik, N. (2001). Clarifying misconceptions between technology education and educational technology, The Technology Teacher 61(1): 31-35. Dumas, J. R. and Redish, J. C. (1993). A practical guide to usability testing, Technical report, Norwood, NJ, Ablex Publishing Corporation. Dunham, T., Wells, J. and White, K. (2002). Biotechnology education: A multiple instructional strategies approach, Journal of Technology Education JTE 14(1). Accessed: 14/Jan/2004. Ebling, M. R. and John, B. E. (2000). On the contributions of different empirical data in usability testing, Conference proceedings on Designing interactive systems : processes, practices, methods, and techniques, ACM Press, pp. 289-296. eduTools (2002). Compare All Products by All Features, http : //www. edutools inf o/ course/compare/all . j sp. Accessed: 1/May/2004. Entwistle, N. and Ramsden, R. (1983). Understanding Student Learning, Croom Helm, London. Ericsson, K. A. and Simon, H. (1980). Verbal reports as data), Psychological Review 87(3): 215-251. Etgen, M. and Cantor, J. (1999). What does getting WET (web event-logging tool) mean for web usability, Proceedings of the Fifth Conference on Human Factors and the Web, Gaithersburg, MD. Ewing, J. and Miller, D. (2002). A framework for evaluating computer supported collaborative learning, Educational Technology and Society 5(1): 112-118. Feizabadi, S. (1996). History of the World Wide Web, Technical report, Virginia Polytechnic Institute and State University. Accessed: 11/Jan/2004. Ferguson, J. D., McGettrick, A. D. and Smeed, D. (1999). Quality Issues in a Virtual University, Computers and Advanced Technology in Education, Proceedings CATE99, Philadelphia, pp. 8-12. Ferguson, J., Miller, J. and Weir, G. (1999). Estimating the Number of Defects at a WWW Site, World Conference on the WWW and Internet (WebNet), Honolulu, Hawaii. Finck, N. (1999). Form vs. Function: Finding the Balance, Digital Web Magazine. http : //www . digital-web . com/f eatures/f eature_1999-10 shtml. FirstClass (2004). FirstClass, http : //www. centrinity. . com/AboutFC/. Accessed: 6/Mar/2004. Flanders, V. and Willis, M. (1998). Web pages that suck : Learn good design by looking at bad design, Sybex, Alameda, CA. Fleming, J. (1998). Web Navigation: Designing the User Experience, Technical report, Sebastopol, CA, O'Reilly and Associates. Florins, M. and Vanderdonckt, J. (2004). Graceful degradation of user interfaces as a design method for multiplatform systems, Proceedings of the 9th international conference on Intelligent user interface, ACM Press, pp. 140-147. Francis, B., Kauffman, J., Llibre, J., Sussman, D. and Ullman, C. (1998). Beginning Active Server Pages 2.0., Wrox Press, Birmingham. Frederiksen, J. R. and White, B. Y. (1997). Cognitive Facilitation: A Method for Promoting Reflective Collaboration, Proceedings of the Second International Conference on Computer Support for Collaborative Learning, Toronto, pp. 53-62. Fritsch, M. (1997). Platforms for Virtual Seminars: Comparison of the features of the three internet-communication-tools, http : //www f ernuni-hagen . de/ZIFF/ vergleichE . html. Accessed: 7/Feb/2004. Gagne, R., Briggs, L. and Wager, W. (1992). Principles of Instructional Design, 4th edition edn, HBJ College Publishers, Fort Worth. TX. Garzotto, F. and Matera, M. (1997). A Systematic Method for Hypermedia Usability Inspection, The New Review of Hypermedia and Multimedia 3: 39-65. George, P. (2000). Breaking ranks, Principal Leadership 1(4): 56-61. Goddard, T. and Sunderam, V. S. (1999). Toolspace: Web based 3d collaboration, Proceedings of the fourth symposium on Virtual reality modeling language, ACM Press, pp. 161-166. Goodman, B., Soller, A., Linton, F. and Gaimari, R. (1998). Encouraging Student Reflection and Articulation using a Learning Companion, International Journal of Artificial Intelligence in Education 9: 237-255. Goodwin, N. C. (1987). Functionality and usability, Communications of the ACM 30(3): 229-233. Gromov, G. R. (1995). History of Internet and WWW: The Roads and Crossroads of Internet History, http: //www .netvalley. com/intvalold html. Accessed: 12/Mar/2004. Grudin, J. (1994). Groupware and social dynamics: eight challenges for developers, Communications of the ACM 37(1): 92-105. Grudin, J. (2002). Group dynamics and ubiquitous computing, Commun. ACM 45(12): 74-78. Guillemette, R. A. (1995). Human factors in information systems: Emerging theoretical bases, C. J. M. (Ed.). Norwood, NJ, Ablex, chapter The evaluation of usability in interactive information systems, pp. 207-221. Gunn, C. (1997). CAL Evaluation: future directions, ALT-J 5(1): 40-47. Halstead, M. H. (1977). Elements of software science, Elsevier, New York. Hammond, M. (1994). Measuring the impact of IT on learning, Journal of Computer Assisted Learning 10: 251-260. Hammond, N. and Allinson, L. (1998). Travels around a learning support environment: Rambling, orienteering or touring, CHI'88 - Conference on Human Factors in Computing Systems, New York. Harasim, L., Hiltz, S. R., Teles, L. and Turoff, M. (1995). Learning Networks: A Field Guide to Teaching and Learning Online, MIT Press, Cambridge MA. Heer, J. and Chi, E. H. (2002). Separating the swarm: categorization methods for user sessions on the web, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 243-250. Herder, E. (2002). Metrics for the adaptation of site structure, Proc. of the German Workshop on Adaptivity and User Modeling in Interactive Systems ABIS02, pp. 22- 26. Hiltz, S. R. (1994). The Virtual Classroom: Learning without limits via computer networks, Ablex Publishing Corporation, New Jersey. Holtzblatt, K. and Beyer, H. (1993). Making customer-centered design work for teams, Communications of the ACM 36(10): 92-103. Huang, Y.-W., Huang, S.-K., Lin, T.-P. and Tsai, C.-H. (2003). Web application security assessment by fault injection and behavior monitoring, Proceedings of the twelfth international conference on World Wide Web, ACM Press, pp. 148-159. IEEE1061 (1992). IEEE 1061 Standard for a Software Quality Metrics Methodology, Report. IS013407 (1999). International Organization for Standardization, ISO/IEC. 13407 Human-Centred Design Processes for Interactive Systems, Report. IS014598-5 (1998). International Organization for Standardization, ISO 14598-5 Information technology – Software product evaluation -- Part 5: Process for evaluators, Report. IS09126 (1991). International Organization for Standardization, ISO/IEC 9126. Information Technology - Software Product Evaluation - Quality characteristics and guidelines for their use, Report. IS09241 (1998). International Organization for Standardization, ISO 9241-11 Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability, Report. Ivory, M. Y. and Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces, ACM Comput. Surv. 33(4): 470-516. Ivory, M. Y., Sinha, R. R. and Hearst, M. A. (2001a). Empirically validated web page design metrics, Proceedings of the SIG CHI conference on Human factors in computing systems, ACM Press, pp. 53-60. Ivory, M. Y., Sinha, R. R. and Hearst, M. A. (2001b). Empirically validated web page design metrics, Proceedings of the SIG CHI conference on Human factors in computing systems, ACM Press, pp. 53-60. Janssen, C., Weisbecker, A. and Ziegler, J. (1993). Generating user interfaces from data models and dialogue net specifications, INTERCHI'93. Jeffries, R., Miller, J. R., Warton, C. and Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques, Human factors in computing systems conference proceedings on Reaching through technology, ACM Press, pp. 119-124. Johnson, D. W. and Johnson, R. (1994). Joining together: group theory and skills, 5th ed. edn, Prentice Hall, Englewood Cliffs, New Jersey. Jokela, T., Iivari, N., Matero, J. and Karukka, M. (2003). The standard of user-centered design and the standard definition of usability: analyzing iso 13407 against iso 9241- 11, Proceedings of the Latin American conference on Human-computer interaction, ACM Press, pp. 53-60. Jonassen, D. (1994). Towards a constructivist design model, Educational Technology 34(4): 34-37. Jonassen, D. H., Beissner, K. and Yacci, M. A. (1993). Structural knowledge: Techniques for representing, conveying, and acquiring structural knowledge, Hillsdale, NJ, Lawrence Erlbaum. Jonassen, D. and Li, T. (1996). Modeling Justification and Reflection in Case-Based Learning Environments, Proceedings of the International Conference on the Learning Sciences (ILLS' 96), pp. 159-163. Jones, A., Scanlon, E., Tosunoglu, C., Ross, S., Butcher, P., Murphy, P. and Greenberg, J. (1996). Evaluating CAL at the Open University: 15 Years On, Computers in Education 26(1-3): 5-15. Jorgensen, A. H. (1989). Designing and Using Human-Computer Interfaces and Knowledge Based Systems. G. Salvendy and M. J. Smith., Amsterdam, Elsevier Science Publishers, chapter Using the thinking-aloud method in system development, pp. 743-750. Kahn, M. J. and Prail, A. (1994). Formal usability inspections, John Wiley & Sons, Inc., pp. 141-171. Kaptelinin, V., Nardi, B. and Macaulay, C. (1999). The Activity Checklist: A Tool for Representing the "Space" of Context, Interactions pp. 27--39. Karat, C.-M., Campbell, R. and Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation, Conference proceedings on Human factors in computing systems, ACM Press, pp. 397-404. Kearsly, G., Seidel, R. and Park, D. K. (1993). Theory Into Practice. A hypertext Database for Learning and Instruction, US Army Research Institute. Kim, W. C. and Foley, J. D. (1993). Providing High-level Control and Expert Assistance in the User Interface Presentation Design, Proceedings 1NTERCHI'93: Human Factors in Computing Systems, pp. 430-437. Kock, N., Sallis, P. J., Masi, C., Auspitz, C. and King, B. (2000). A Rule-Based Method for Determining the Degree of Student Satisfaction of a Web-Based Learning Environment, Americas Conference on Information Systems (AIS). Kopetzky, T. and Muhlhauser, M. (1999). Visual Preview for Link Traversal on the World Wide Web, Computer Networks 31: 1525-1532. Koschmann, T. (1995). CSCL: Theory and Practice of an Emerging Paradigm (Computers, Cognition, and Work), Lawrence Erlabaum, Hillsdale. NJ. Kubilus, N. J. (2000). Designing an e-commerce site for users, Crossroads 7(1): 23-26. Kumari, S. (1996). Teaching with the Internet, Conference proceedings of The Second International N.A. Web Conference. Accessed: 12/Jan/2004. Larson, K. and Czerwinski, M. (1998). Web page design: implications of memory, structure and scent for information retrieval, Conference proceedings on Human factors in computing systems, ACM Press/Addison-Wesley Publishing Co., pp. 25- 32. Laurillard, D. (1993). Rethinking University Teaching: a framework for the effective use of educational technology, London, Routledge. Laurillard, D. (1995). Multimedia and the changing experience of the learner, British Journal of Educational Technology 26(3): 179-189. Law, L.-C. and Hvannberg, E. T. (2002). Complementarity and convergence of heuristic evaluation and usability test: a case study of universal brokerage platform, Proceedings of the second Nordic conference on Human-computer interaction, ACM Press, pp. 71-80. LearningSpace (2004). LearningSpace, http: //www. lotus . com/products/ learnspace nsf/. Accessed: 6/Mar/2004. Lee, S. H. (1999). Usability Testing for Developing Effective Interactive Multimedia Software: Concepts, Dimensions, and Procedures, Educational Technology Society 2(2): 20-28. Leont'ev, A. N. (1978). Activity, Consciousness, Personality, Englewood Cliffs, NJ, Prentice Hall. Leont'ev, A. N. (1981). Problems of the Development of Mind, Moscow: Progress Publishers. Levi, M. D. and Conrad, F. G. (1996). A Heuristic Evaluation of a World Wide Web Prototype, Interactions 3(4): 51-61. Levi, M. D. and Conrad, F. G. (1997). Usability Testing of World Wide Web Sites: A CHI 97 Workshop, SIGCHI Bulletin 29(4): 40-43. Lewis, C. (1982). Using the 'thinking-aloud' method in cognitive interface design, Yorktown Heights NY, T J Watson Research Centre. Lewis, C., Poison, P. G., Wharton, C. and Rieman, J. (1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces, Conference proceedings on Empowering people : Human factors in computing system: special issue of the SIGCHI Bulletin, ACM Press, pp. 235-242. Lewis, J. R. (1994). Sample sizes for usability studies: Additional considerations, Human Factors 36: 368-378. Lindgaard, G. (1994). Usability testing and system evaluation), London, Chapman & Hall. Lindquist, T. E. (1985). Assessing the Usability of Human-Compute Interface, IEEE Software pp. 74-82. Lohse, G. and Spiller, P. (1999). Internet retail store design: How the user interface influences traffic and sales, Journal of Computer-Mediated Communication. (JCMC) 5(2). www.ascusc.org/jcmc/vol5/issue2/. Lowgren, J. and Nordquist, T. (1992). Knowledge-Based Evaluation as Design Support for Graphical User Interfaces, Conference proceedings on Human factors in computing systems, pp. 181-188. Lund, K., Baker, M. J. and Baron, M. (1996). Modelling Dialogue and Beliefs as a Basis for Generating Guidance in a CSCL Environment, ITS-96 Conference, Montreal. Lynch, P. and Horton, S. (1997). Yale Style Manual-Balanced page designs, http: //info .med yale edu/caim/manual/contents . html. Accessed: 12/Jun/2004. Mahajan, R. and Shneiderman, B. (1997). Visual and Textual Consistency Checking Tools for Graphical User Interfaces, IEEE Transactions on Software Engineering 23(11): 722-735. Mao, J.-Y., Vredenburg, K., Smith, P. W. and Carey, T. (2001). User-centered design methods in practice: a survey of the state of the art, Proceedings of the 2001 conference of the Centre for Advanced Studies on Collaborative research, IBM Press, p. 12. Marchionini, G. and Shneiderman, B. (1988). Finding facts vs. browsing knowledge in hypertext system, IEEE Computer 21(3): 70-80. McCabe, T. J. (1976). A Complexity measure, IEEE Transactions on Software Engineering SE-2(4): 308-320. McCabe, T. J. and Butler, C. W. (1989). Design Complexity Measurement and Testing, Communications of the ACM 32(12): 1415-1425. McDonald, S. and Stevenson, R. (1998). Navigation in Hyperspace: An Evaluation of the Effects of Navigational Tools and Subject Matter Expertise on Browsing and Information Retrieval in Hypertext, Interacting with Computers 10: 129-142. McEneaney, J. E. (2001). Graphic and numerical methods to assess navigation in hypertext, Journal of Human-Computer Studies 55: 761-786. McManus, T. F. (1996). Delivering Instruction on the World Wide Web, http : //www.svsu.edu/-mcmanus/papers/wbi . html. Accessed: 17/Jan/2004. Medlock, M. C., Wixon, D., Terrano, M., Romero, R. and Fulton, B. (2002). Using the RITE Method to improve products: a definition and a case study, Usability Professionals Association, ACM Press. Misic, M. and Johnson, K. (1999). Benchmarking: a tool for Web site evaluation and improvement, Internet Research: Electronic Networking Applications and Policy 9(5): 383-392. Molich, R., Thomsen, A. D., Karyukina, B., Schmidt, L., Ede, M., van Oel, W. and Arcuri, M. (1999). Comparative evaluation of usability tests, CHI '99 extended abstracts on Human factors in computing systems, ACM Press, pp. 83-84. Monk, A., Wright, P., Haber, J. and Davenport, L. (1993). Improving Your Human- Computer Interface: A Practical Technique, Hemel Hempstead, U.K., Prentice Hall International. MU (2000). Marshall University's: Comparison of Online Course Delivery Software Products, http : //www .marshall edu/it/cit/webct/compare/ comparison . html. Accessed: 18/Jan/2004. Muller, M. J., Dayton, T. and Root, R. (1993). Comparing studies that compare usability assessment methods: an unsuccessful search for stable criteria, INTERACT '93 and CHI '93 conference companion on Human factors in computing systems, ACM Press, pp. 185-186. Newell, A. and Simon, H. A. (1972). Human Problem Solving, Englewood Cliffs, NJ, Prentice-Hall. Niederst, J. (1999). Web Design in a Nutshell: A Desktop Quick Reference, Tokyo , O'Reilly. Nielsen, J. (1989). Usability engineering at a discount, Proceedings of the third international conference on human-computer interaction on Designing and using humancomputer interfaces and knowledge based systems (2nd ed.), Elsevier Science Inc., pp. 394-401. Nielsen, J. (1990). Evaluating Hypertext Usability. Designing Hypermedia for Learning, D. H. Jonassen, and Mandl, H., Springer-Verlag Berlin Heidelberg. Nielsen, J. (1992a). Advances in Human-Computer Interaction, Vol. 3, H. R. Hartson and D. Hix. Norwood, NJ, Ablex, chapter Evaluating the thinking aloud technique for use by computer scientists, pp. 69-82. Nielsen, J. (1992b). The usability engineering life cycle, Computer 25(3): 12-22. Nielsen, J. (1993). Usability Engineering, Boston, MA, Academic Press, New York. Nielsen, J. (1994a). Cost-Justifying Usability R.G. Bias and, D.J. Mayhew (Eds), Academic Press, chapter Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier. http : //www . use it . com/papers/guerrilla_ hc . html. Nielsen, J. (1994b). Enhancing the explanatory power of usability heuristics, Conference proceedings on Human factors in computing systems : "celebrating interdependence", ACM Press, pp. 152-158. Nielsen, J. (1994c). Usability inspection methods, Proceedings of the CHI '94 conference companion on Human factors in computing systems, ACM Press, pp. 413-414. Nielsen, J. (1995). Usability inspection methods, Conference companion on Human factors in computing systems, ACM Press, pp. 377-378. Nielsen, J. (1998). Cost of User Testing a Website, http : //www. use it . com/alertbox/ 980503 . html. Accessed: 12/Feb/2004. Nielsen, J. (1999a). The Top Ten New Mistakes of Web Design, http : //www . useit . com/alertbox/990530,htm1. Accessed: 12/May/2004. Nielsen, J. (1999b). User Interface Directions for the Web, Communications of the ACM 42(1): 65-72. Nielsen, J. (2000a). Severity Ratings for Usability Problems, http : //www. use it . com/ papers/heurist ic/severityrat ing html. Accessed: 7/Mar/2004. Nielsen, J. (2000b). Why You Only Need to Test With 5 Users, http : //www.useit com/alertbox/20000319 . html. Accessed: 8/Mar/2004. Nielsen, J. (2001a). Success Rate: The Simplest Usability Metric, http //www useit com/alertbox/20010218 html. Accessed: 8/Mar/2004. Nielsen, J. (2001b). Usability Metrics, http : //www. use it . com/alertbox/20010121. html. Accessed: 12/Feb/2004. Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems, ACM INTERCHI, Amsterdam, the Netherlands. Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces, Conference proceedings on Empowering people : Human factors in computing system: special issue of the SIGCHI Bulletin, ACM Press, pp. 249-256. Nielsen, J. and Phillips, V. L. (1993). Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared, Proceedings of the conference on Human factors in computing systems, Addison-Wesley Longman Publishing Co., Inc., pp. 214-221. Nielsen, J. and Wagner, A. (1996). User Interface Design for the WWW, CHI '96 Conference, Vancouver, British Columbia. of Manitoba, U. (2001). Tools for Developing Interactive Academic Web Courses, http : //www .umanitoba. ca/ip/tools/courseware/evalmain html. Accessed: 1/May/2004. Oliver, K., Wilkinson, G. and Bennett, L. (1997). Evaluating the Quality of Internet Information Sources, The Annual Convention of the Association for the Advancement of Computing in Education (AACE),ED-MEDIA/ED-TELECOM 97. Olsina, L. and Rossi, G. (1999). Toward Web-site Quantitative Evaluation: Defining Quality Characteristics and Attributes, World Conference on the WWW and Internet (WebNet). Honolulu, Hawaii. Paloff, R. and Pratt, K. (1999). Building Learning Communities in Cyberspace, San Francisco: Jossey-Bass. Park, J. and Kim, J. (2000). Effects of contextual navigation aids on browsing diverse web systems, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 257-264. Parush, A., Nadir, R. and Shtub, A. (1998). Evaluating the layout of graphical user interface screens: Validation of a numerical, computerizedmodel, International Journal of Human Computer Interaction 10(4): 343360. Petschenik, M. H. (1985). Practical priorities in system testing, IEEE Software pp. 18- 23. Polson, P. G., Lewis, C., Rieman, J. and Wharton, C. (1992). Cognitive walkthroughs: a method for theory-based evaluation of user interfaces, International Journal of Man-Machine Studies 36(5): 741-773. Prates, R. 0., de Souza, C. S. and Barbosa, S. D. J. (2000). Methods and tools: a method for evaluating the communicability of user interfaces, interactions 7(1): 31- 38. Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S. and T.Carey (1994). Humancomputer interaction, Workingham, England, Addison-Wesley. Pulat, B. M. and Nwankwo, H. H. (1987). Formatting alphanumeric CRT displays, International Journal of Man-Machine Studies 26(5): 567-580. Ramsden, P. (1992). Learning to Teach in Higher education, London, Rountledge. Ratner, J., Grose, E. M. and Forsythe, C. (1996). Characterization and assessment of HTML style guides, Proceedings of the CHI '96 conference companion on Human factors in computing systems : common ground, ACM Press, pp. 115-116. Rauterberg, M. (1992). In: Van der Veer, G., Tauber, M, e.a.: Human-Computer Interaction: Tasks and Organisation, CUD-Publ, Roma, chapter A Method of Quantitative Measurement of Cognitive Complexity, pp. 295-307. Ruberg, L. F., More, D. M. and Taylor, D. (1996). Student Participation, Interaction and Regulation in a Computer Mediated Communciation Environment: A Qualitative Study, Educational Computer Research 14(3): 243-268. Rubin, J. (1994). Handbook of Usability Testing, New York, NY,John Wiley and Sons. Saljo, R. (1979). Learning in the learner's perpective. Some common-sense conceptions, The Institute of Education, University of Gothenburg. Sa 's, A. J., Bygrave, J. and Carran, G. (2000). Developing and Evaluating An On-line Cross-Cultural Collaborative Course, World Conference on Educational Multimedia, Hypermedia & Telecommunications (ED-MEDIA), Montreal, Canada. Savard, M., Mitchell, S. N., Abrami, P. C. and Corso, M. (1995). Learning Together at a Distance, Canadian Journal of Educational Educaction 24(2): 117-131. Sawyer, B. (1996). The ultimate web developer's source book, AZ,The Coriolis Group, Inc. Sawyer, P., Flanders, A. and Wixon, D. (1996). Making a difference the impact of inspections, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 376-382. Schmeck, R. (1981). Improving learning by improving thinking, Educational Leadership 38: 384-385. Scholtz, J. and Laskowski, S. (1998). Developing usability tools and techniques for designing and testing web sites, Proceedings of the Fourth Conference on Human Factors and the Web. Sears, A. (1993). Layout appropriateness: A metric for evaluating user interface widget layout, IEEE Transactions on Software Engineering 19(7): 707-719. Sears, A. (1995). AIDE: A Step Towards Metric Based Development Tools, 8th ACM Symposium on User Interface and Software Technology, Pittsburg, PA. Shackel, B. (1991). Human factors for informatics usability, B.S. S. J. R. (Eds.). Cambridge, Cambridge University, chapter Usability-Context, framework, definition, design and evaluation, pp. 21-37. Shahabi, C. and Zarkesh, A. (1997). Knowledge Discovery from Users Web-page Navigation, 7th Int. Workshop on Research Issues in Data Eng. on High Performance Database Management for Large-Scale Applications, IEEE Comp. Soc. Press. Shang, Y., Hongchi, S. and Su-Shing, C. (2001). An intelligent distributed environment for active learning, Journal of Educational Resources in Computing (JERIC) 1(2es): 4. Shepperd, M. and Ince, D. (1990). Multi-dimensional modelling and measurement of software designs, Proceedings of the 1990 ACM annual conference on Cooperation, ACM Press, pp. 76-81. Sherkat, F. (2002). Deep Learning through online delivery of Technological Courses:A quest for teaching excellence, Proceedings of The Eighth Australian World Wide Web Conference. Shneiderman, B. (1997). Designing Information-abundant Web Sites: Issues and Recommendations, International Journal of Human-Computer Studies 47(1): 5-29. Shneiderman, B., Chimera, R., Jog, N., Stimart, R. and White, D. (1997). Display Systems: Design and Applications Edited by L. MacDonald and A. Lowe, U.K. John Wiley & Sons. Shneiderman, B. and Kearsley, G. (1989). Hypertext Hands-On! An Introduction to a New Way of Organizing and Accessing Information, Addison-Wesley. Simonson, M., Smaldino, S., Albright, M. and Zvacek, S. (2003). Teaching and learning at a distance (2nd ed.), Upper Saddle River, NJ: Merrill Prentice Hall. Srnerdon, B., Cronen, S., Lanahan, L., Anderson, J., Iannotti, N. and Angeles, J. (2000). Teachers' tools for the 21st century: A report on teachers' use of technology, Technical report, U.S. Department of Education, Washington, D.C.:National Center for Education Statistics. Smith, P. (1996). Towards a practical measure of hypertext usability, Interacting with Computers 8(4): 365-381. Smyth, B. and Cotter, P. (2002). The Plight of the Navigator: Solving the Navigation Problem for Wireless Portals, Proceedings of the Second International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Springer-Verlag, pp. 328-337. Spool, J. M., Schroeder, W., Scanlon, T. and Snyder, C. (1999). Web sites that work: designing with your eyes open, CHI '99 extended abstracts on Human factors in computing systems, ACM Press, pp. 141-142. Spool, J. and Schroeder, W. (2001). Testing Web Sites: Five Users is Nowhere Near Enough, Proceedings of ACM CHI Conference on Human Factors in Computing, Seattle, WA, USA. *citeseer nj .nec.com/spoo101 testing.html Standards, N. I. 0. and Technology (2001). NIST, http://zing.ncsl.nist .gov/ WebTools/WebSAT/form_use.shtml. Accessed: 18/Mar/2004. Stern, M. K. (1997). The Difficulties in Web-Based Tutoring, and Some Possible Solutions. Intelligent Educational Systems on the World Wide Web, 8th World Conference of the AIED Society, Kobe, Japan. Steves, M. P., Morse, E., Gutwin, C. and Greenberg, S. (2001). A comparison of usage evaluation and inspection methods for assessing groupware usability, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, ACM Press, pp. 125-134. Stilborne, L. (1996). The SchoolNet Digital Collections Project: Building a Knowledge Base on the Web, N.A. WEB 96, The University of New Brunswick, Canada. Stotts, P. D. and Furuta, R. (1989). Petri-net-based hypertext: document structure with browsing semantics, ACM Transactions on Information Systems (TOIS) 7(1): 3-29. Stotts, P., Furuta, R. and Ruiz, J. (1992). Hyperdocuments as automata: Trace-based browsing property verification, Conference proceedings of Hypertext (ECHT '92), ACM Press, pp. 272-281. Suebnukarn, S. and Haddawy, P. (2004). A collaborative intelligent tutoring system for medical problem-based learning, Proceedings of the 9th international conference on Intelligent user interface, ACM Press, pp. 14-21. Szekely, P., Luo, P. and Neches, R. (1992). Facilitating the exploration of interface design alternatives: the HUMANOID model of interface design, Conference proceedings on Human factors in computing systems, ACM Press, pp. 507-515. Tagg, A. C. (1994). Leadership from Within: Student Moderation of Computer Conferences, American Journal of Distance Education 8(3): 40-50. Tarasewich, P. (2000). Designing Usable Web Sites, WebNet Journal 2(2): 23-30. Taylor, S. J. E. (2000). Groupware and the simulation consultant, Proceedings of the 32nd conference on Winter simulation, Society for Computer Simulation International, pp. 83-89. Thomas, P. and Macredie, R. D. (2002). Introduction to the new usability, ACM Trans. Comput.-Hum. Interact. 9(2): 69-73. Tielemans, G. and Collis, B. (1999). Strategic Requirements for a System to Generate and Support WWW-Based Environments for a Faculty, World Conference on Educational Multimedia, Hypermedia & Telecommunications (ED-Media), Seattle, USA. Tognazzini, B. (1998). First Principles, http : //www asktog . com/basics/ f irstPrinciples html. Accessed: 17/Jun/2004. Tullis, T. S. (1983). The formatting of alphanumeric displays: A review and analysis, Human Factors 25(6): 657-682. Tullis, T. S. (1988). Advances in Human-Computer Interaction, H. Rex Hartson, and Hix, D. Norwood, New Jersey, Ablex Publishing, chapter A System or Evaluating Screen Formats: Research and Application, pp. 214-286. Turnwald, G. H., Bull, K. S. and Seeler, D. C. (1993). From Teaching to Learning: Part II. Traditional Teaching Methodology, Journal of Veterinary Medical Education 20(3). http://scholar.lib.vt.edu/ejournals/JVME/V20-3/turnwald.html. Tyran, C. K. and George, J. F. (2002). Improving software inspections with group process support, Commun. ACM 45(9): 87-92. UAT (2001). University of Austin Texas - Blackboard vs WebCT, http : //www . utexas . edu/cc/blackboard/about/atut/evalIntro.html. Accessed: 15/Jan/2004. van der Veen, J., Boer, W. F. and van de Ven, M. (2000). W3LS: Evaluation framework for World Wide Web learning, IEEE Journal for Educational Technology Society 3(4): 132-138. Vanderdonckt, J. and Gillo, X. (1994). Visual techniques for traditional and multimedia layouts, Proceedings of the workshop on Advanced visual interfaces, ACM Press, pp. 95-104. Vanderdonckt, J. M. and Bodart, F. (1993). Encapsulating knowledge for intelligent automatic interaction objects selection, Proceedings of the conference on Human factors in computing systems, Addison-Wesley Longman Publishing Co., Inc., pp. 424- 429. Virzi, R. (1990). Streamlining the design process: Running fewer subjects, Proceedings of the Human Factors Society 34th annual meeting, pp. 291-294. Virzi, R. (1992). Refining the test phase of usability evaluation: How many subjects is enough?, Human Factors 34: 457-468. W3C (1999). World Wide Web Consortium: Web Content Accessibility Guidelines 1.0, WWW:http : //www .w3 . org/TR/WAI-WEBCONTENT/. Accessed: 13/Aug/2004. Watkins, D. A. (1983). Depth of processing and the quality of learning outcomes, Instructional Science pp. 49-58. WebCT (2004). WebCT, http : //www. webct . com/. Last Accessed: 5/Jan/2004. Weber, G. and Brusilovsky, P. (2001). ELM-ART: An adaptive versatile system for Web-based instruction, International Journal of Artificial Intelligence in Education 12(4): 351-384. Weber, G. and Specht, M. (1997). User Modeling and Adaptive Navigation Support in WWW-Based Tutoring Systems, Proceedings of the Sixth International Conference on User Modeling. http : //w5 . cs uni-sb de/-UM97/abstracts/WeberG. html. Wertsch, J. (1981). The Concept of Activity in Soviet Psychology, Sharpe. Wharton, C., Rieman, J., Lewis, C. and Poison, P. (1994). The cognitive walkthrough method: a practitioner's guide, John Wiley & Sons, Inc., pp. 105-140. Williams, R. and Tollett, J. (2000). The Non-Designer's Web Book, 2nd edition edn, Peachpit Press. Wright, P. and Lickorish, A. (1989). In R. McAleese, C. Green (Eds.) Hypertext: State of the art, Oxford: Intellect, chapter An empirical comparison of two navigation systems for two hypertexts, pp. 84-93. Zheng, Y. and Pong, M. (1992). Using statecharts to model hypertexts, In Lucarella, D., editor, Proc. ECHT92, pp. 242-250. Zirkle, C. and Shoemaker, H. (1999). Indiana State's multiple delivery approach: Integrating industrial technology education with educational technology, The Technology Source . Retrieved March 06, 2004, from http : //www .mivu. org/teaching/ techsource/. Adrion, W. R., Barnstad, M. and Cherniaysky, J. (1982). Validation, verification, and testing computer software, ACM Computing Surveys 14(2): 159-192. AL-Qaimari, G. and McRostie, D. (1999). KALDI: A computer-aided usability engineering tool for supporting testing and analysis of humancomputer interaction., Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces, Kluwer Academic Publishers. Albrecht, A. J. (1985). Function points help manager assess applications, Computerworld (SR/20): 20-21. Anderson, C., Domingos, P. and Weld, D. (2001). Personalizing Web Sites for Mobile Users, Proceedings of the 10th International WWW Conference, ACM Press, pp. 565-575. Anderson, R., Homer, A., Sussman, D. and Blexrud, C. (1999). Professional Active Server Pages 3.0., Birmingham, Wrox Press Inc. Arroyo, I., Beck, J., Beal, C. R. and Woolf, B. (2003). Learning within the Zone of Proximal Development with the Animal Watch intelligent tutoring system, Proceedings of the American Educational Research Association annual meeting. Accessed: 5/ Mar/ 2004. ASU (2000). Arizona State University - 10 Critical Success Factors for Cyber Campuses, http://clte.asu.edu/active/Critical%20Success%20Factors.doc. Accessed: 16/Feb/2002. Atkins, M. (1993). Evaluating interactive technologies for learning, Journal of Curriculum Studies 25(4): 333-342. Babiker, E. M., Fujihara, H. and Boyle, C. D. B. (1991). A metric for hypertext usability, Proceedings of the 9th annual international conference on Systems documentation, ACM Press, pp. 95-104. Bannon, L. (1991). Design at Work: Cooperative Design of Computer Systems. J. Greenbaum e4 M.Kyng(Eds.), Hillsdale, NJ: Lawrence Erlbaum Associates, chapter From human factors to human actors: The role of psychology and human-computer interaction studies in system design, pp. 25-44. Basili, V. and Perricone, B. T. (1984). Software Errors and Complexity: An Empirical Investigation, Communications of the ACM 27(1): 42-52. Beal, C. R., Arroyo, I., Royer, J. and Woolf, B. (2003). Wayang Outpost: An intelligent multimedia tutor for high stakes math achievement tests, Proceedings of the American Educational Research Association annual meeting. Accessed: 5/Mar/2004. Becker, R., McLaughlin, K. and Rebelsky, S. (1999). Project CLIO: Tools for tracking student use of course webs, World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA, Seattle, USA. Benbunan-Fich, R. (2002). Improving education and training with IT, Communications of the ACM 45(6): 94-99. Berkman, J. (1997). Web Design and Maintenance, 25th SIGUCCS Conference on User Services, Monterey, CA. Berners-Lee, T. (1990). WorldWideWeb: Proposal for a HyperText Project, http : //www w3 org/Proposal . html. Accessed: 11/Jun/2003. Bevan, N. (1999). Quality in use: meeting user needs for quality, Journal of Systems and Software 49(1): 89-96. Bevan, N., Barnum, C., Cockton, G., Nielsen, J., Spool, J. and Wixon, D. (2003). The "magic number 5": is it enough for web testing?, CHI '03 extended abstracts on Human factors in computing systems, ACM Press, pp. 698-699. Bias, R. G. (1994). The pluralistic usability walkthrough: coordinated empathies, John Wiley and Sons, Inc., pp. 63-76. Biggs, J. B. and Telfer, R. (1987). The Process of Learning, 2 edn, Sydney, Prentice- Hall. Billsus, D., Brunk, C. A., Evans, C., Gladish, B. and Pazzani, M. (2002). Adaptive interfaces for ubiquitous web access, Communications of the ACM 45(5): 34-38. Blackboard (2004). Blackboard, http: //www . blackboard . cm/. Accessed: 5/Jan/2004. Blackmon, M. H., Polson, P. G., Kitajima, M. and Lewis, C. (2002). Cognitive walkthrough for the web, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 463-470. Bodart, F., Hennebert, A.-M., Leheureux, J.-M. and Vanderdonckt, J. (1994). Towards a Dynamic Strategy for Computer-Aided Visual Placement, 2nd Workshop on Advanced Visual Interfaces, Bari, Italy. Bodker, S., Ehn, P., Knudsen, J., Kyng, M. and Madsen, K. (1988). Computer support for cooperative design (invited paper), Proceedings of the conference on Computersupported cooperative work, ACM Press, pp. 377-394. Borges, J. A., Morales, I. and Rodrguez, N. J. (1996). Guidelines for Designing Usable World Wide Web Pages, Proceedings of the CHI '96 conference companion on Human factors in computing systems: common ground, ACM Press, pp. 277-278. http://doi.acm.org/10.1145/257089.257320. Botafogo, R. A., Rivlin, E. and Shneiderman, B. (1992). Structural analysis of hypertexts: identifying hierarchies and useful metrics, ACM Transactions on Information Systems (TOIS) 10(2): 142-180. Broder, A. and Kumar, R. (2000). Graph structure in the web, Proceedings of the 9th Intl. WWW Conference, ACM Press, pp. 309-320. Brown, A. and Palinscsar, A. (1989). In Lauren B. Resnick (Ed.) Knowledge, Learning and Instruction, Lawrence Erlbaum, Hillsdale, NJ, chapter Guided, Cooperative Learning and Individual Knowledge Acquisition. Brown, J. and Duguid, P. (1994). Borderline issues: Social and material aspects of design, Human-Computer Interaction 9(1): 3-36. Brusilovsky, P. and Maybury, M. T. (2002). From adaptive hypermedia to the adaptive web, Commun. ACM 45(5): 30-33. Brusilovsky, P., Ritter, S. and Schwarz, E. (1997). Distributed intelligent tutoring on the Web, 8th World Conference on Artificial Intelligence in Education, Amsterdam, IOS. Brusilovsky, P. and Rizzo, R. (2002). Map-based horizontal navigation in educational hypertext, Proceedings of the thirteenth ACM conference on Hypertext and hypermedia, ACM Press, pp. 1-10. Buie, E. (1999). HCI standards: a mixed blessing, Interactions 6(2): 36-42. Bygrave, J., Sallis, A., Matheny, J. and Carran, G. (1999). Collaborative Learning Online in International Business, Southeast Asian Institutions of Higher Learning (Asaihl), Auckland, New Zealand. Byrne, E. (2000). Evaluate Web Resources, http : //www. clubi .ie/webserch/ resources/content . htm. Accessed: 21/Dec/2003. C2T2 (2001). Online educational delivery applications: a web tool for comparative analysis, http: //www. c2t 2 . ca/. Accessed: 12/Jan/2004. Canale, R. (1998). Review of Web based, Course Delivery Software Systems, http: //www. dls . au. com/papers/review-cds html. Accessed: 1/May/2004. Center, 0. S. (2002). Tools, Reviews and Comparisons, http: //www.osc edu/ education/webed/Tools/review. shtml. Accessed: 8/Feb/2003. CERN (1998). The world wide web, CERN, http: //public . web . cern. ch/public/ about/achievements/www/www.html. Accessed: 12/Jan/2004. CERN (2002). History and growth, CERN, http : //public . web . cern. ch/public/ about/achievements/www/history/history . html. Accessed: 15/Jan/2004. Chakrabarti, S., Gibson, D. and McCurley, K. (1999). Surfing the Web Backwards, Computer Networks 31: 1679-1693. Chi, E. H., Rosien, A., Supattanasiri, G., Williams, A., Royer, C., Chow, C., Robles, E., Dalai, B., Chen, J. and Cousins, S. (2003). The bloodhound project: automating discovery of web usability issues using the InfoScentπ simulator, Proceedings of the conference on Human factors in computing systems, ACM Press, pp. 505-512. Chi, M. T. H., Bassok, M., Lewis, M. W., Reinmann, P. and Galser, R. (1989). Self- Explanations: How Students Study and Use Examples in Learning to Solve Problems, Cognitive Science 13: 145-482. Choudhary, P. (1998). A Comparative Analysis Of Online Education Tools, http : //www . vis c . vt . edu/succeed/wwwframework/survey html. Accessed: 14/Dec/2002. Cohen, E. (1994). Restructuring the classroom: Conditions for productive small groups, Review of Educational Research 64(1): 1-35. Coll, R. and Wingertsman, J. C. (1990). The effect of screen complexity on user preference and performance, International Journal of Human-Computer Interaction 2(3): 255-265. Collins, A. (1990). Educational Values and Cognitive Instruction, Lawrence Erlbaum, Hillsdale. NJ, chapter Cognitive Apprenticeship and Instructional Technology, In Idol, L. and Jones, B. F. (Eds.). Comber, T. and Maltby, J. (1996). Computer-aided design of user interfaces, J. Vanderdonckt. Namur, Belgium, Press Universitaries de Namur, chapter Investigating layout complexity, pp. 209-227. Conklin, J. (1987). Hypertext: An introduction and survey, IEEE Computer 20(9): 17– 41. Constantine, L. L. and Lockwood, L. A. D. (2002). Usage-centered engineering for web applications, IEEE Software 19(2): 42-50. http //www. computer. org/software/ so2002/s2toc . htm. Conte, S. D., Dunsmore, H. E. and Shen, V. Y. (1986). Software Engineering Metrics and Models, Benjamin/Cummings Publishing Company, Inc., Menlo Park, California. Corry, M., Frick, T. W. and Hansen, L. (1997). User-centered design and usability testing of a web site: An illustrative case study, Educational Technology, Research and Development 45(4): 65-76. http : //education. Indiana. edu/ -edfolks/persons/frick.html. Coulston, C. and Vitolo, I. M. (2001). A hypertext metric based on huffman coding, Proceedings of the twelfth ACM conference on Hypertext and Hypermedia, ACM Press, pp. 243-244. Crampes, M. and Ranwez, S. (2000). Ontology-supported and ontology-driven conceptual navigation on the world wide web, Proceedings of the eleventh ACM on Hypertext and hypermedia, ACM Press, pp. 191-199. Darken, R. P. and Sibert, J. (1996). Navigating in Large Virtual Worlds, International Journal of Human-Computer Interaction 8(1): 49-72. De Angell, A., Matera, M., Costabile, M. F., Garzotto, F. and Paolini, P. (2000). Validating the sue inspection technique, Proceedings of the working conference on Advanced visual interfaces, ACM Press, pp. 143-150. DeBra, P., Houben, G., Kornatzky, Y. and Post, R. (1994). Information retrieval in distributed hypertexts, 4th RIAO-94 Conference, New York. DeMarco, T. (1982). Controlling Software Projects: Management, Measurement and Estimation, Yourdon Press, New York. Denaro, G. and Pezz, M. (2002). An empirical evaluation of fault-proneness models, Proceedings of the 2/4 th international conference on Software engineering, ACM Press, pp. 241--251. der Aalst, W. V., Bra, P. D., Houben, G. J. and Kornatzky, Y. (1993). Computing Science Notes 93, chapter Browsing semantics in the "tower" model, p. 47. Desurvire, H. W. (1994). Faster, cheaper!! Are usability inspection methods as effective as empirical testing?, John Wiley and Sons, Inc., pp. 173-202. Desurvire, H. W., Kondziela, J. M. and Atwood, M. E. (1993). What is gained and lost when using evaluation methods other than empirical testing, Proceedings of the conference on People and computers VII, Cambridge University Press, pp. 89-102. Dicks, R. S. (2002). Mis-usability: on the uses and misuses of usability testing, Proceedings of the 20th annual international conference on Computer documentation, ACM Press, pp. 26-30. DiNucci, D., Giudice, M. and Stiles, L. (1998). Elements of web design, Technical report, Berkley CA, PeachPit Press. Dix, A., Finlay, J., Abowd, G. and Beale, R. (1993). Human-Computer Interaction, 2nd edn, London, Prentice Hall. Doubleday, A., Ryan, M., Springett, M. and Sutcliffe, A. (1997). A comparison of usability techniques for evaluating design, Proceedings of the conference on Designing interactive systems, ACM Press, pp. 101-110. Draper, S. (1998). Niche-based success in CAL, Computers and Education 30(1/2): 5- 8. Draper, S., Brown, M., Henderson, F. and McAteer, E. (1997). Integrative Evaluation: An Emerging Role for Classroom Studies of CAL, Computers in Education 26(1- 3): 17-32. Dugger, W. and Naik, N. (2001). Clarifying misconceptions between technology education and educational technology, The Technology Teacher 61(1): 31-35. Dumas, J. R. and Redish, J. C. (1993). A practical guide to usability testing, Technical report, Norwood, NJ, Ablex Publishing Corporation. Dunham, T., Wells, J. and White, K. (2002). Biotechnology education: A multiple instructional strategies approach, Journal of Technology Education JTE 14(1). Accessed: 14/Jan/2004. Ebling, M. R. and John, B. E. (2000). On the contributions of different empirical data in usability testing, Conference proceedings on Designing interactive systems processes, practices, methods, and techniques, ACM Press, pp. 289-296. eduTools (2002). Compare All Products by All Features, http : //www edutools inf o/ course/compare/all . j sp. Accessed: 1/May/2004. Entwistle, N. and Ramsden, R. (1983). Understanding Student Learning, Croom Helm, London. Ericsson, K. A. and Simon, H. (1980). Verbal reports as data), Psychological Review 87(3): 215-251. Etgen, M. and Cantor, J. (1999). What does getting WET (web event-logging tool) mean for web usability, Proceedings of the Fifth Conference on Human Factors and the Web, Gaithersburg, MD. Ewing, J. and Miller, D. (2002). A framework for evaluating computer supported collaborative learning, Educational Technology and Society 5(1): 112-118. Feizabadi, S. (1996). History of the World Wide Web, Technical report, Virginia Polytechnic Institute and State University. Accessed: 11/Jan/2004. Ferguson, J. D., McGettrick, A. D. and Smeed, D. (1999). Quality Issues in a Virtual University, Computers and Advanced Technology in Education, Proceedings CATE99, Philadelphia, pp. 8-12. Ferguson, J., Miller, J. and Weir, G. (1999). Estimating the Number of Defects at a WWW Site, World Conference on the WWW and Internet (WebNet), Honolulu, Hawaii. Finck, N. (1999). Form vs. Function: Finding the Balance, Digital Web Magazine. http://www.digital-web.com/features/feature_1999-10.shtml. FirstClass (2004). FirstClass, http //www. centrinity. . com/AboutFC/. Accessed: 6/Mar/2004. Flanders, V. and Willis, M. (1998). Web pages that suck : Learn good design by looking at bad design, Sybex, Alameda, CA. Fleming, J. (1998). Web Navigation: Designing the User Experience, Technical report, Sebastopol, CA, O'Reilly and Associates. Florins, M. and Vanderdonckt, J. (2004). Graceful degradation of user interfaces as a design method for multiplatform systems, Proceedings of the 9th international conference on Intelligent user interface, ACM Press, pp. 140-147. Francis, B., Kauffman, J., Llibre, J., Sussman, D. and Ullman, C. (1998). Beginning Active Server Pages 2.0., Wrox Press, Birmingham. Frederiksen, J. R. and White, B. Y. (1997). Cognitive Facilitation: A Method for Promoting Reflective Collaboration, Proceedings of the Second International Conference on Computer Support for Collaborative Learning, Toronto, pp. 53-62. Fritsch, M. (1997). Platforms for Virtual Seminars: Comparison of the features of the three internet-communication-tools, http : //www f ernuni-hagen . de/ZIFF/ vergleichE . html. Accessed: 7/Feb/2004. Gagne, R., Briggs, L. and Wager, W. (1992). Principles of Instructional Design, 4th edition edn, HBJ College Publishers, Fort Worth. TX. Garzotto, F. and Matera, M. (1997). A Systematic Method for Hypermedia Usability Inspection, The New Review of Hypermedia and Multimedia 3: 39-65. George, P. (2000). Breaking ranks, Principal Leadership 1(4): 56-61. Goddard, T. and Sunderam, V. S. (1999). Toolspace: Web based 3d collaboration, Proceedings of the fourth symposium on Virtual reality modeling language, ACM Press, pp. 161-166. Goodman, B., Soller, A., Linton, F. and Gaimari, R. (1998). Encouraging Student Reflection and Articulation using a Learning Companion, International Journal of Artificial Intelligence in Education 9: 237-255. Goodwin, N. C. (1987). Functionality and usability, Communications of the ACM 30(3): 229-233. Gromov, G. R. (1995). History of Internet and WWW: The Roads and Crossroads of Internet History, http: //www .netvalley. com/intvalold html. Accessed: 12/Mar/2004. Grudin, J. (1994). Groupware and social dynamics: eight challenges for developers, Communications of the ACM 37(1): 92-105. Grudin, J. (2002). Group dynamics and ubiquitous computing, Commun. ACM 45(12): 74-78. Guillemette, R. A. (1995). Human factors in information systems: Emerging theoretical bases, C. J. M. (Ed.). Norwood, NJ, Ablex, chapter The evaluation of usability in interactive information systems, pp. 207-221. Gunn, C. (1997). CAL Evaluation: future directions, ALT-J 5(1): 40-47. Halstead, M. H. (1977). Elements of software science, Elsevier, New York. Hammond, M. (1994). Measuring the impact of IT on learning, Journal of Computer Assisted Learning 10: 251-260. Hammond, N. and Allinson, L. (1998). Travels around a learning support environment: Rambling, orienteering or touring, CHI'88 - Conference on Human Factors in Computing Systems, New York. Harasim, L., Hiltz, S. R., Teles, L. and Turoff, M. (1995). Learning Networks: A Field Guide to Teaching and Learning Online, MIT Press, Cambridge MA. Heer, J. and Chi, E. H. (2002). Separating the swarm: categorization methods for user sessions on the web, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 243-250. Herder, E. (2002). Metrics for the adaptation of site structure, Proc. of the German Workshop on Adaptivity and User Modeling in Interactive Systems ABIS02, pp. 22- 26. Hiltz, S. R. (1994). The Virtual Classroom: Learning without limits via computer networks, Ablex Publishing Corporation, New Jersey. Holtzblatt, K. and Beyer, H. (1993). Making customer-centered design work for teams, Communications of the ACM 36(10): 92-103. Huang, Y.-W., Huang, S.-K., Lin, T.-P. and Tsai, C.-H. (2003). Web application security assessment by fault injection and behavior monitoring, Proceedings of the twelfth international conference on World Wide Web, ACM Press, pp. 148-159. IEEE1061 (1992). IEEE 1061 Standard for a Software Quality Metrics Methodology, Report. IS013407 (1999). International Organization for Standardization, ISO/IEC. 13407 Human-Centred Design Processes for Interactive Systems, Report. IS014598-5 (1998). International Organization for Standardization, ISO 14598-5 Information technology — Software product evaluation -- Part 5: Process for evaluators, Report. IS09126 (1991). International Organization for Standardization, ISO/IEC 9126. Information Technology - Software Product Evaluation - Quality characteristics and guidelines for their use, Report. IS09241 (1998). International Organization for Standardization, ISO 9241-11 Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability, Report. Ivory, M. Y. and Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces, ACM Comput. Surv. 33(4): 470-516. Ivory, M. Y., Sinha, R. R. and Hearst, M. A. (2001a). Empirically validated web page design metrics, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 53-60. Ivory, M. Y., Sinha, R. R. and Hearst, M. A. (2001b). Empirically validated web page design metrics, Proceedings of the SIG CHI conference on Human factors in computing systems, ACM Press, pp. 53-60. Janssen, C., Weisbecker, A. and Ziegler, J. (1993). Generating user interfaces from data models and dialogue net specifications, INTERCHI'93. Jeffries, R., Miller, J. R., Warton, C. and Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques, Human factors in computing systems conference proceedings on Reaching through technology, ACM Press, pp. 119-124. Johnson, D. W. and Johnson, R. (1994). Joining together: group theory and skills, 5th ed. edn, Prentice Hall, Englewood Cliffs, New Jersey. Jokela, T., Iivari, N., Matero, J. and Karukka, M. (2003). The standard of user-centered design and the standard definition of usability: analyzing iso 13407 against iso 9241- 11, Proceedings of the Latin American conference on Human-computer interaction, ACM Press, pp. 53-60. Jonassen, D. (1994). Towards a constructivist design model, Educational Technology 34(4): 34-37. Jonassen, D. H., Beissner, K. and Yacci, M. A. (1993). Structural knowledge: Techniques for representing, conveying, and acquiring structural knowledge, Hillsdale, NJ, Lawrence Erlbaum. Jonassen, D. and Li, T. (1996). Modeling Justification and Reflection in Case-Based Learning Environments, Proceedings of the International Conference on the Learning Sciences (ILLS' 96), pp. 159-163. Jones, A., Scanlon, E., Tosunoglu, C., Ross, S., Butcher, P., Murphy, P. and Greenberg, J. (1996). Evaluating CAL at the Open University: 15 Years On, Computers in Education 26(1-3): 5-15. Jorgensen, A. H. (1989). Designing and Using Human-Computer Interfaces and Knowledge Based Systems. G. Salvendy and M. J. Smith., Amsterdam, Elsevier Science Publishers, chapter Using the thinking-aloud method in system development, pp. 743-750. Kahn, M. J. and Prail, A. (1994). Formal usability inspections, John Wiley & Sons, Inc., pp. 141-171. Kaptelinin, V., Nardi, B. and Macaulay, C. (1999). The Activity Checklist: A Tool for Representing the "Space" of Context, Interactions pp. 27--39. Karat, C.-M., Campbelen_NZ
 Find in your library

Files in this item

FilesSizeFormatView

There are no files associated with this item.

This item is not available in full-text via OUR Archive.

If you would like to read this item, please apply for an inter-library loan from the University of Otago via your local library.

If you are the author of this item, please contact us if you wish to discuss making the full text publicly available.

This item appears in the following Collection(s)

Show simple item record