ISQua's 39th International Conference 2023

Validation of the Rates of Adverse Event Incidence in Administrative Healthcare Data through Patient Chart Review: A Scoping Review
Anna Connolly, Marcia Kirwan and Anne Matthews
School of Nursing, Psychotherapy and Community Health, Dublin City University

Introduction

Reporting and monitoring of adverse events is key as patient safety is a growing public health challenge and a significant healthcare quality concern. Routinely collected hospital discharge data, has been recognised as a valuable source of information on adverse events and a potentially efficient tool for monitoring patient safety in hospital settings (Raleigh et al., 2008). The challenges associated with using administrative data for research purposes have been highlighted previously (McGuckin et al., 2022), particularly in relation to poor recording of adverse events (Kim et al., 2022). Retrospective chart review is commonly used in healthcare-based disciplines to assess healthcare quality and inpatient care (Vasser and Holzmann, 2013) and is a reliable way of identifying whether administrative data accurately reflects the events of a patient’s stay in hospital (Nissen et al., 2019). The rationale for conducting this scoping review is underpinned by the various calls for validation of diagnostic codes used to translate information within patients’ chart into administrative data (Mazzali and Duca, 2015; Raleigh et al., 2008; van Mourik et al., 2015; Ehrenstein et al., 2016).

Aims

1.To present an overview of how chart review has been used by previous researchers to validate rates of adverse events in administrative data.

2.To identify research gaps, chart the evidence of and highlight any flaws within administrative datasets to improve extraction and coding practices and enable researchers and policy makers to use these data to their full potential

Methods

Figure 1: Methodology and Review Process. (A) This scoping review was conducted in line with the Joanna Briggs Institute (JBI) methodology for scoping reviews (Peters et al., 2020). (B) The sources identified through the database searches were imported into Covidence. Duplicates were removed, identified sources were screened and included or excluded based on the inclusion/exclusion criteria. The remaining 57 studies were included in the review and had their data extracted using Covidence

Results

Figure 2: Demographic Study Details (A) Distribution of included studies according to year of publication. (B) Distribution of included studies according to geographical location of publication. (C) Distribution of included studies according to specialty

Figure 3: Study Design. Various named study designs were used in the included studies therefore, only the most commonly named study designs have been graphed above. The remaining studies each had their own unique named study design.

Figure 4: Strategy Used to Identify Sample of Charts for Review. Screening programmes were used in 24 studies to identify the chart review sample by flagging charts that had been assigned specific codes, therefore indicating that the patient had experienced an adverse event. In these studies, only the flagged charts were included in the chart review. A further 11 studies adopted a case-control design approach and included both flagged charts and non-flagged charts in their review. The remaining 22 studies selected a random sample of charts that were not flagged with any indication that the patient had or had not experienced an adverse event.

Figure 5: Measured Accuracy of Recording of Adverse Events in Administrative Data. Some studies reported on the accuracy of the codes themselves and other reported on the accuracy of screening programmes based on administrative data however, for the purpose of this review accuracy will be looked at collectively in relation to these two groups. A significant proportion of studies reported that the rates of adverse events were poorly recorded in administrative data. The potential of administrative data to record adverse events with moderate accuracy was outlined by several studies. Many studies that reported moderate accuracy suggested that administrative data can be useful for monitoring patient safety and identifying potential cases of patient safety incidents but not for public reporting or measuring performance. Fewer studies identified good accuracy of administrative data for recording adverse events however, these studies highlight the potential of using administrative data for research purposes. It is also important to note that two studies (Campbell, 2011 and Reilly, 2020) that reported poor accuracy of administrative data also identified the chart review methodology as being poor at detecting adverse events.

Discussion and Conclusion

•The upward trend in relation to year of publication demonstrates the increasing recognition of the chart review methodology as a tool for conducting validation studies on rates of adverse events in administrative data.

•A significant proportion of the studies were conducted in the United States (52%), suggesting that further exploration of this method of validating administrative data is required globally.

•Although chart reviewing was most commonly used to validate rates of surgical adverse events, rates of adverse events relating to various specialties were validated. This demonstrates the potential of the chart review methodology to validate rates of various types of adverse events in administrative data.

•The various named study designs used in the included studies demonstrates the lack of a standardised named study design to describe the use of the chart review to validate rates of adverse events.

•Whilst some studies relied on a random sampling strategy to validate administrative data by capturing all possible adverse events as recorded in the patients’ charts, the chart reviews that were conducted on flagged datasets had the potential to miss out on recording adverse events that had not been coded in the charts in the first place.

•Although various levels of accuracy at capturing adverse events was reported, administrative data has the potential to provide accurate information. Whilst some studies indicated that these datasets can be reliable, others identified inaccurate adverse event incidence represented in administrative datasets when compared with data recorded in patients’ clinical charts.

•This review highlights the potential for administrative datasets to improve patient safety, healthcare quality and reduce costs to both patients and healthcare systems by providing valuable data on adverse event rates however, further work is warranted to ensure that administrative data is robust and accurate.

References

Ehrenstein, V., Petersen, I., Smeeth, L., Jick, S., Benchimol, E.I., Ludvigsson, J.F., Sørensen, H.T. (2016) ‘Helping everyone do better: A call for validation studies of routinely recorded health data’, Clinical Epidemiology, 8, pp. 49-51

Kim, J., Choi, E. Y., Lee, W., Oh, H. M., Pyo, J., Ock, M., Kim, S. Y. and Lee, S. I. (2022) ‘Feasibility of Capturing Adverse Events From Insurance Claims Data Using International Classification of Diseases, Tenth Revision, Codes Coupled to Present on Admission Indicators’, Journal of patient safety, 18(5), pp. 404–409.

Mazzali, C., Duca, P. (2015) ‘Use of administrative data in healthcare research’, Internal and Emergency Medicine, 10, pp. 517–524. Doi: 10.1007/s11739-015-1213-9

McGuckin T, Crick K, Myroniuk TW, Setchell, B., Yeung, R. O. and Cambell-Scherer, D. (2022) ‘Understanding challenges of using routinely collected health data to address clinical care gaps: a case study in Alberta, Canada’, BMJ Open Quality, 11(e001491) doi:10.1136/ bmjoq-2021-00149

Nissen, F., Quint, J.K., Morales, D.R., Douglas, I.J. (2019) ‘How to validate a diagnosis recorded in Electronic Health Records’, Breathe, 15(1), pp. 64–68.

Peters, M. D. J, Godfrey, C., McInerney, P., Munn, Z., Tricco, A. C. and Khalil, H. (2020) ‘Chapter 11: Scoping Reviews’, in Aromataris, E. and Munn, Z. (eds.) JBI Manual for Evidence Synthesis, JBI, 2020. Available at: https://doi.org/10.46658/JBIMES-20-12

Raleigh, V. S., Cooper, J., Bremner, S. A. and Scobie, S. (2008) ‘Patient safety indicators for England from hospital administrative data: case-control analysis and comparison with US data’, BMJ (Clinical research ed.), 337, a1702. doi: 10.1136/bmj.a1702

van Mourik, M. S., van Duijn, P. J., Moons, K. G., Bonten, M. J., and Lee, G. M. (2015) ‘Accuracy of administrative data for surveillance of healthcare-associated infections: a systematic review’, BMJ open, 5(8), e008424. Doi: 10.1136/bmjopen-2015-008424

Vassar, M. and Matthew, H. (2013) ‘The retrospective chart review: Important methodological considerations’, Journal of Educational Evaluation for Health Professions, 10(12) doi:10.3352/jeehp.2013.10.12.

ISQua's 39th International Conference 2023

VALIDATION OF THE RATES OF ADVERSE EVENT INCIDENCE IN ADMINISTRATIVE HEALTHCARE DATA THROUGH PATIENT CHART REVIEW: A SCOPING REVIEW

Poster Presenter

Authors

Affiliations

Part of Topic