ISQua's 40th International Conference 2024

Impact of on-site audits on the evaluation of European Reference Networks for the care of patients with rare diseases

P. PALOMERA RUFO1, I. DE ULIBARRI GROCIN1, I. M. ALONSO PORTO1, M. SAINZ RUIZ1 and A. SAN NICOLÁS MEDINA2

1 IDOM Consulting, Engineering, Architecture

2 Instituto de Excelencia Europea

Topic: External Evaluation

P647

INTRODUCTION:

ERNs are virtual networks of clinical units whose mission is to share knowledge to improve access to specialized and quality care for European patients with rare or complex diseases. Decision 2014/287/EU(1) establishes a five-year evaluation of the performance and contribution to the care of these patients by the 24 ERNs, and the 836 units or Healthcare Providers (HCPs) belonging to these networks.

The evaluation methodology (2,3) includes an initial self-evaluation by HCPs, followed by a documentary review and on-site audits performed by an evaluation team.

AIM:

The main goal is to understand the impact of on-site audits on a randomized sample of 193 Healthcare Providers. This was achieved by a thorough analysis and comparison of the data and results obtained before and after the conduction of the on-site audits and comparing results of audited and non-audited Healthcare Providers.

METHOD:

1.Random selection of 193 HCPs using an in-house developed computer code considering ERN-level and geographical representativeness, and ERN suggestions.

2.Assigning evaluators to the different centres usingan in-house developed computer code that considers evaluators’ and HCPs’ availability, possible conflicts of interest with the hospital or ERN,and the evaluators’ role (whether they were a Team Leader or a Team Member).

3.Training for evaluators and audited centres.

4.Support for the execution of different audit phases: review of medical records, interviews with professionals and patients.

5.Compilation and comparison of results obtained before and after the audits.

6. Analysis of the results and comments obtained from satisfaction surveys.

RESULTS:

•38.4% of audited HCPs and 8.8% of the total evaluated HCPs improved their result from Needs Improvement to Satisfactory after the on-site audit.

•170 of the audited HCPs obtained a Satisfactory result in the evaluation.

•Only 1 out of the 24 ERNs did not change the number Satisfactory audited HCPs after the onsite audits.

•The areas that improved the most after the on-site audits were "Patient-Centred Care" and "Quality and Safety“.

•Over 70% of evaluators and HCPs are satisfied with the methodology and development of the on-site audits.

CONCLUSIONS:

•The audit process has positively impacted the evaluation results of the audited Healthcare Providers.

•The audits help to assess evidence more effectively.

•36.9% of evaluable criteria should be adapted for non-audited Healthcare Providers.

•All parts have expressed their satisfaction with the methodology and development of the audits.

•There's a need to address ethical or confidentiality issues with the Healthcare Providers' ethical committees prior to audits.

•There's a need to differentiate between management and bureaucratic issues, for the hospital management team, and patient care issues, for the medical team.

Several aspects to improve the on-site audits were drawn from the satisfaction surveys from both evaluators and HCPs:

•There should be at least one visit per hospital.

•The methodology to choose the medical records to review and the patients to interview should be reviewed.

•Considering the added value of the onsite-audits, online interviews should be scheduled for non-audited HCPs.

•More time should be given to both evaluators and Healthcare Providers to properly prepare for the on-site audit.

•The questions of the audit should be less administrative and more focused on patient care.

ACKNOWLEDGEMENT:

Our heartfelt gratitude to the HCPs and ERNs for their invaluable support and commitment throughout the evaluation process. Their dedication has been essential in ensuring its success.

Our sincere thanks to the European Commission for entrusting us with the responsibility of carrying out this important evaluation and its constant support. Also, to the evaluators for their outstanding work and dedication throughout the entirety of the project.

REFERENCES:

1 COMMISSION DELEGATED DECISION of 10 March 2014 setting out criteria and conditions that European Reference Networks and healthcare providers wishing to join a European Reference Network must fulfil. OJ L 147, 17.5.2014, p. 79–87

2 Technical Toolbox: overview of Annexes for the Evaluation process [Internet]. [cited 2024 Sep 6]. Available from: https://health.ec.europa.eu/document/download/d8fe72ce-94dd-4f98-8311-2ea7ab3b4e2d_en?filename=erns_evaluation_toolbox_en.pdf

‌3 EVALUATION MANUAL [Internet]. [cited 2024 Sep 6]. Available from: https://health.ec.europa.eu/document/download/4cff3905-7e55-46b3-8fb6-4e62a3d263f4_en?filename=erns_evaluation_manual_en.pdf

CONTACT INFORMATION:

Patricia Palomera Rufo

ppalomerar@idom.com

+34 676 273 247

ISQua's 40th International Conference 2024

IMPACT OF ON-SITE AUDITS ON THE EVALUATION OF EUROPEAN REFERENCE NETWORKS FOR THE CARE OF PATIENTS WITH RARE DISEASES

Poster Presenter

Authors

Affiliations

Part of Topic