An Evaluation of Legal Information Chatbots: Useability, Utility, and Accuracy

Bridgette Toy-Cronin; Mihiata Pirini; David Turner; Dawn Duncan

This report was prepared for Citizen AI, Community Law Wellington (Citizen AI’s owner), and Community Law Centres O Aotearoa to provide information about the useability and utility of a Citizen AI developed legal information chatbot, Rentbot. It also includes an expert evaluation of the accuracy of a second Citizen AI chatbot, Workbot. These chatbots—Rentbot and Workbot—provide legal information on tenancy and employment matters respectively and are currently available online. Chatbots are a promising development in the delivery of legal information to the public. They are potentially a means to providing a more targeted and dynamic delivery of legal information, while avoiding the inevitable expense of one-to-one services. Citizen AI’s development of these bots is an important step forward for legal information in Aotearoa New Zealand. Our testing found some issues with finding Rentbot, although this may, at least in part, be explained by our testing methods. Rentbot—once users have found the tool—proved useful. While it did make a number of retrieval errors and one minor legal error in the tests, participants were generally positive about its performance. It was stronger in its performance with the legally simpler of the two scenarios we tested with users, as it did not reliably give all the information required for the second scenario. That said, it outperformed what was available in an internet search and, therefore, is useful in the sense of providing more information than is otherwise available to a user. It frequently, although not always, provided users with a pathway to a next step in solving their problems and users were generally positive about their interactions and the interface. Workbot is a “younger” bot operating in a more complex area of law. The testing we conducted for this bot was also more limited. The testing that was conducted indicated some concerns around its accurate retrieval of information and the need to ensure retrieved information was appropriate. It also required a level of accuracy in search terms (spelling and language) which may be unrealistic for the average user. We consider there are a range of factors that should be taken into account when considering future planning: the costs of maintaining and updating the bots as the law changes; better integration into the legal information landscape (including Community Law’s own manual) to ensure users can find the bots; and using the bots as a platform to guide users in their next steps in solving their legal issues. Reflecting on these issues will help guide decisions on building from the considerable effort and ingenuity that has gone into the bots’ creation.

An Evaluation of Legal Information Chatbots: Useability, Utility, and Accuracy

(1)