Norms are an important component of the social fabric of society by
prescribing expected behaviour. In Multi-Agent Systems (MAS), agents
interacting within a society are equipped to possess social capabilities such
as reasoning about norms and trust. Norms have long been of interest within the
Normative Multi-Agent Systems community with researchers studying topics such
as norm emergence, norm violation detection and sanctioning. However, these
studies have some limitations: they are often limited to simple domains, norms
have been represented using a variety of representations with no standard
approach emerging, and the symbolic reasoning mechanisms generally used may
suffer from a lack of extensibility and robustness. In contrast, Large Language
Models (LLMs) offer opportunities to discover and reason about norms across a
large range of social situations. This paper evaluates the capability of LLMs
to detecting norm violations. Based on simulated data from 80 stories in a
household context, with varying complexities, we investigated whether 10 norms
are violated. For our evaluations we first obtained the ground truth from three
human evaluators for each story. Then, the majority result was compared against
the results from three well-known LLM models (Llama 2 7B, Mixtral 7B and
ChatGPT-4). Our results show the promise of ChatGPT-4 for detecting norm
violations, with Mixtral some distance behind. Also, we identify areas where
these models perform poorly and discuss implications for future work.
- 9926553294701891
- Norm Violation Detection in Multi-Agent Systems using Large Language Models: A Pilot Study
- Shawn HeSurangika RanathungaStephen CranefieldBastin Tony Roy Savarimuthu
- arXiv.org
- School of Computing
- English
- Preprint