Abstract
Background
The introduction and roll-out of the COVID-19 vaccine was one of the most important and complex initiatives undertaken by health sectors world wide. In New Zealand the Education, Training and Clinical Support to the health sector of the immunisation programme was undertaken by the Immunisation Advisory Centre (IMAC). An embedded evaluation was planned from the outset, using a range of scientific approaches to acknowledge the complexity of the COVID-19 immunisation context.
The aim was to act as a rigorous 'critical friend' to the programme: appreciating and recording successes, identifying challenges and opportunities early, and working with stakeholders to generate innovative solutions (Hindsight to Foresight).
Method
The evaluation used an established methodology and evidence base combining complexity and implementation science with appreciative inquiry (CIS-A), previously applied in immunisation strategy and mental health settings.
Both numerical and narrative data were collected to provide information to the programme for endorsement of what was going well, and the early identification of challenges. Methods included analysis of routinely collected workforce data, help line call data, surveys, focus groups and interviews with vaccinators, vaccinees and key stakeholders. There was also direct observation of vaccination sites and settings.
Results
The embedded evaluation process was sustained throughout the COVID-19 immunisation response. Regular assessment and feedback enabled recognition and support of positive features of activity information about challenges with the opportunity for early course correction.
The evaluation was able to assess the overall success and contribution of the programme to the immunisation rollout.
There were specific areas where external evaluation was especially helpful, the focus on important and concise "Toolkit messages for the day" approach and the focus on equity being two examples.
One of the challenges in using an iterative embedded methodology is that what might have been traditional 'end of project' evaluation findings have already been raised and, in many cases, acted upon. Changes have therefore already been made and there may be a perception that an evaluation is adding little to project development.
Conclusions
We recommend that using a multi perspective embedded evaluation approach can be of value in implementing complex health sector initiatives.