Abstract
The multi-criteria decision-making (MCDM) method known as PAPRIKA, as implemented in the 1000minds application, has been in use since 2004. This thesis addresses the research problems that arise in attempting to improve PAPRIKA’s algorithms and how to analyse its outputs. These improvements are intended to address three main problems that people have encountered since its release: how to measure agreement of rankings of alternatives, how to produce scores with ratio-scale measurement properties, and how to reduce the burden of making tradeoffs.
Measuring agreement
A common mistake in comparing points systems (models) by calculating correlations of the alternatives to be ranked (e.g. patients, projects, houses) is reviewed, demonstrating that such correlations are usually spurious. After pursuing various other means of assessing agreement between models, a general purpose tool is developed to calculate significance levels for correlations of all hypothetical alternatives. The significance calculator is applied to literature in an area of health economics, reassessing authors' conclusions and highlighting examples of mistaking spurious correlations as meaningful associations.
Ratio-scale measurement
Most MCDM methods calculate scores (as well as ranks) for alternatives. Until now, the scores produced by PAPRIKA were considered to have at least ordinal-scale measurement properties, yet these scores are frequently interpreted as having ratio-scale measurement properties, despite the lack of a theoretical basis and with stark examples to the contrary. After demonstrating its foundations in measurement theory, PAPRIKA is supplemented with a means of assessing uncertainty, which can be used to report progress towards arriving at scores with interval-scale measurement properties. Another process is created to help decision-makers find a point of zero value (fundamental to any ratio scale), thereby arriving at scores with quasi ratio-scale measurement properties. This new process is used to create a New Zealand version of the EQ-5D-5L – a health-related quality of life tool.
Decision-maker burden
The PAPRIKA method repeatedly poses questions that require a tradeoff between competing criteria. Larger models require more tradeoffs, making the process more time-consuming for decision-makers and increasing the likelihood of their making mistakes. The use of impossible combinations of levels of criteria (e.g. it may be considered impossible for someone to be in extreme pain and to be able to undertake their usual activities) is reviewed and a new simpler method for applying them to reduce the number of questions is introduced.
Next, the reason why humans sort with fewer comparisons, on average, than a computer, is explained, and various options for reducing the expected number of tradeoffs by shrewd selection of questions to pose to the decision-maker are proposed and tested. The best-performing method, dubbed PAPRIKA⁺⁺, reduces the average number of tradeoffs required by over 20% in some cases. PAPRIKA⁺⁺ is implemented and tested in decision-making surveys alongside the original PAPRIKA, demonstrating its effectiveness for real people.