Data-Driven Evaluation of Residents’ Clinical Competence:Automating The Model of Clinical Practice of Emergency Medicine
To build a tool to automatically track resident clinical encounters by mapping all items of the Model of Clinical Practice of EM to diagnostic and procedural codes already recorded in patient charts.
•Accurate, efficient tracking of procedures has been the subject of recent research and innovation. Procedural competency is only a portion of the knowledge and experience residents need to obtain through postgraduate training, however.
•The Model of Clinical Practice of EM (“EM Model”) is a comprehensive list of everything the fully-trained EM physician should have mastered. It serves as the basis for ABEM certification exams and curricular planning by major national organizations such as ACEP, SAEM, and CORD-EM.
•In 2011, Tintinalli et al published a study quantifying the variation in clinical encounters among trainees in the same program. They found substantial variation among residents in the same cohort, 30% to 60%, with maximal variation corresponding to roughly 1 year of clinical training.
•There is currently no accepted method for tracking residents’ clinical encounters, or their progress through the EM Model.
•We are mapping each EM Model item to 1 or more ICD-10 codes, and each procedure to 1 or more CPT (Current Procedural Terminology) codes.
•These surrogates are nearly universal searchable constants in the EMR.
•There are 68,000 ICD-10-CM codes used for diagnoses, and 87,000 ICD-10-PCS codes for procedures. There are approximately 775 discrete items listed in the EM Model (excluding “other core competencies” such as interpersonal and communication skills, professionalism, etc.)
•The map, or index, of codes and items serves as the basis for a tool that can be programmed to run within the EMR, capturing data on every clinical encounter documented by every resident.
•This tool can generate thorough empiric data on an individual resident’s progress.
•Data can be aggregated to provide an overview of an entire residency cohort’s clinical exposures, revealing common deficiencies.
•Accurate, automated tracking via the EMR can provide powerful data demonstrating how residents’ clinical work experience compares to the full scope of Emergency Medicine.
•This data can identify deficiencies to be addressed in curriculum planning for the program and for individual residents. Specifically, it could help guide:
•patient selection on shift
•the development of individualized learning plans
•conference lecture planning
•simulation case development
•evaluation of required rotations, clinical sites
•choice of electives
Testing the tool
•In order to test the reliability of our tool, we will perform a short, iterative study.
•Resident volunteers will manually track which items of the EM Model their clinical work satisfies during each shift for a given period of time.
•We will then compare this manual record to the report created by the automated tool, and edit the programming of our tool as needed to maximize accurate, reliable capture of all relevant data.
•This study is currently awaiting Institutional Review Board approval.