Abstract
Background: The outcome evaluation of large-scale public health interventions is an important part of the rational policy cycle. Timely outcome evaluation can inform decisions by policy makers on whether to continue, modify or discontinue an intervention. Cost-benefit analysis-based outcome evaluations are particularly valuable, producing results that allow easy comparisons of interventions that have multiple benefits and/or externalities. It is important that the evaluation of benefits and costs be based on sound statistical principles, and correct characterisation of the key features of an intervention is vital if results are to be generalised in order to inform future interventions.
The key element of cost-benefit analyses of large-scale public health interventions is the evaluation of health benefits. Health data present conceptual and analytical challenges due to the characteristics of cost data and the complexities of assigning dollar values to changes in the rates of events such as mortality. Health data from New Zealand’s Warm Up New Zealand: Heat Smart (WUNZ:HS) evaluation, which retrofitted homes with improved insulation and space heating, provided the case study that is the basis of this thesis, enabling the exploration of suitable statistical and econometric approaches to the evaluation of health benefits. A related comparison of the WUNZ:HS programme with Australia’s controversial Home Insulation Programme (HIP) was an opportunity to consider the key characteristics of a successful large-scale public health intervention of this type, an evaluation which complements the outcome evaluation of health cost data.
Aims: The primary aim of this thesis was to examine the methodological issues around the analysis of administrative data in large-scale public health intervention evaluations, using data available from the WUNZ:HS evaluation as the basis of a case study. The methodologies examined aimed to accurately quantify the health benefits or costs of a programme in dollar terms.
Secondary to this aim was a policy analysis-based comparison of the WUNZ:HS programme with the Australian Home Insulation Programme (HIP). This comparison informed retrofit programme specific recommendations and also a discussion about the generalisability of the results of the primary analysis and the importance of intervention design and implementation.
Methods: Mortality data were analysed using Cox Proportional Hazard models. Statistically significant changes in mortality were then valued using a “Value of a Statistical Life”-based methodology. Changes in hospitalisation costs and pharmaceutical use costs were modelled at the household level using difference-in-difference-based fixed effects models. The WUNZ:HS HIP policy comparison was conducted using a combination of a literature review and a key informant interview.
Results: Individuals aged 65 and over who occupied a household that received a WUNZ:HS insulation retrofit and who had had a baseline circulatory hospitalisation had a Hazard Ratio for mortality of 0.673 (95% CI 0.535 - 0.847) (p<0.001) relative to comparable control group individuals. Individuals aged 65 and over who occupied a household that received a WUNZ:HS insulation retrofit and who had had a baseline respiratory hospitalisation had a Hazard Ratio of 0.8 (95% CI 0.635 – 1.008), (p=0.058) relative to comparable control group individuals. Annual household level mortality-related benefits of insulation can be valued at approximately $1,120, although this figure varies greatly depending on modelling assumptions. Heating did not have a statistically significant Hazard Ratio for vulnerable treatment vs. control individuals.
Households that received insulation had reductions in monthly all-cause hospitalisation costs relative to their matched controls of $3.18 per month (p<0.001) (95% CI 1.83 - 4.54) in the favoured model. There was no evidence of an impact of heating retrofits on hospitalisation costs.
Households that received insulation had reductions in monthly all-cause pharmaceutical use costs relative to matched control group households of $2.01 (p<0.001) (95% CI 1.52 – 2.50) in the favoured model. Households that received heating demonstrated reductions in monthly all-cause pharmaceutical use costs of $3.01 (p<0.001) (95% CI 1.43 – 4.59).
The central estimate of the combined annual household health benefits of insulation was $859 using a (p<0.05) threshold for inclusion of benefits and $1,173 using a (p<0.1) threshold. The combined annual household level health benefits of heating were approximately $41 using either threshold. When health results were incorporated into total cost-benefit figures the benefit cost ratio for the programme was estimated at 6.4:1, driven largely by health benefits, indicating strong evidence for the value of the WUNZ:HS programme. This result was highly sensitive to the assumptions used to estimate mortality-related benefits.
Comparison of the WUNZ:HS and HIP programmes suggested that differences in programme outcomes were driven by differences in the characteristics of retrofitting organisations, the financial contribution of households towards retrofits, the experience of implementing agencies, the relative speeds of programme roll-outs, and differences in policy officials’ commitment to economic stimulus vs. other potential programme co-benefits. Some of these differences can be linked to the wider policy contexts, particularly the impact of housing research in the New Zealand context.
Conclusion: Exploration of statistical and econometric methods for the evaluation of large-scale public health interventions, using the WUNZ:HS evaluation dataset as a case study, produced plausible and consistent results, and demonstrated the importance of modelling assumptions. Policy analysis-based comparison of WUNZ:HS and Australia’s HIP demonstrated the importance of characterising key elements of an intervention if results are to be generalisable.