Abstract
Patient safety is the avoidance, prevention and amelioration of adverse outcomes or injuries stemming from the healthcare system. Patient safety is the most important constituent of quality and has become increasing important in relation to hospital care since a series of landmark studies documented the frequency and nature of inpatient incidents that involved patient harm resultant from health care. A number of methods have been proposed to identify and track adverse events including case note review, incident reporting, the observation of care, and questionnaires. However these methods have the limitations that they collect data from non-random and biased populations and are too labour intensive for widespread use. More recently the use of trigger tools has shown some promise as better method to identify harm but concerns still exist about the reliability of the trigger tools and the resources needed to undertake the work. Instead attention has increasingly turned to the use of readily available administrative data that includes comprehensive information about all patient admissions to hospitals coded in an internationally standardised manner. A number of countries have exploited the use of hospital indicators to measure patient safety and among the available sets of indicators the group developed by the Agency for Healthcare Research and Quality in the United States has attracted the most interest. A useful indicator to assess the quality of patient safety should be well defined, feasible, valid and reliable. The AHRQ PSIs indicators were developed by a rigorous process that included empirical evaluation of their reliability and validity. Based on administrative data they are well defined and feasible. Before they can be applied in New Zealand their validity and reliability needs to be determined in the local setting.
Validity is a complicated concept when applied to indicators and includes face validity, content validity, construct validity and criterion validity. The construct validity of the PSIs was assessed in New Zealand by applying the indicators to administrative data and comparing the occurrence of PSI positive admissions with mortality and length of stay. PSI positive admissions were associated with higher 30 day and 90 day mortality rates and longer lengths of inpatient stay. Criterion validity was assessed by evaluating PSI positive admissions against the information included in the medical records of one large New Zealand hospital to confirm whether the definitions required for the indicators were met. The positive predictive values for the PSIs were observed to be at least moderately high (>0.50) for thirteen of the fourteen PSIs. It was concluded that the PSIs were suitably valid and could be applied to New Zealand hospital data.
The AHRQ PSIs were applied to New Zealand hospital data and the variables needed for risk adjusted comparisons of the safety of inpatient care were explored using data from the National Minimum Dataset from the period 2001-2009. Significant relationships between PSI positive admissions and age, sex and comorbidity, measured by the Charlson Comorbidity Index (Charlson, Pompei, Ales, & MacKenzie, 1987), were observed and these variables were selected for risk adjustment models. The relative infrequency of events defined by several PSIs (1, 5, 8, 10, 11, 13 and 16) ensured that some of them were less suitable for use as performance measures across New Zealand hospitals.
Three different types of risk adjustment (AHRQ, hierarchical and propensity score) were explored with five, relatively frequent, PSIs (Death in low mortality DRGs, Failure to rescue, Postoperative haemorrhage or haematoma, postoperative DVT or PE and Accidental puncture or laceration) and all three methods reduced the variability of patient safety indicator results across New Zealand hospitals. The hierarchical risk adjustment method provided the most conservative results and by including hospital level data in conjunction with patient related information offered the largest amount of shrinkage that most successfully reduced the instability of small numbers associated with rare events at small hospitals. The results from the three methods were not consistent and different hospitals were identified by each method as outliers and ranks frequently changed when varying methods were applied. Problems were encountered with all three risk adjustment methods. Convergence issues plagued the hierarchical and AHRQ models while a number of hospitals could not be included in the propensity score method. All three methods were limited by the quality of the administrative data used to generate the measures. Hierarchical risk adjustment is recommended due to its ability to adjust for hospital level as well as patient related factors and conservative results that are less likely to falsely identify outlier hospitals.
Among the five indicators explored by the risk adjustment methods Failure to rescue (PSI 4) appears to be the most promising as a comparative measure of patient safety at New Zealand hospitals. PSI4 provided a relatively stable measure of performance that discriminated between hospitals.
Future work should explore the sensitivity of the indicators and confirm that the events identified are clinical important. Present on admission coding should be integrated in to the indicators and international efforts to harmonise both administrative data and the definitions of the indicators should be supported. The forthcoming arrival of ICD 11 with its enhanced potential to identify adverse events coupled with the imminent arrival of SNOMED CT and shared electronic health records offers unprecedented opportunities to measure and improve patient safety across the whole health system. Improved risk adjustment methods that harvest data from all available sources and undertake analyses with a Bayesian framework will enhance any comparative assessments. Similarly graphical tools like funnel plots and techniques at individual hospitals like cumulative sum control chart (CUSUM) analyses will help with these evaluations. A range of tools and methods will always be needed in order to measure the full spectrum of harm in our health system.