Background

National society guidelines are routinely adapted into institutional protocols for routine clinical guidelines; however these may become outdated and misaligned when new evidence-based recommendations are published. To address this, we built an AI-enabled platform (C8 Health) 
centralizes locally vetted best practices and streamlines dissemination for just-in-time clinical use. We developed and validated an automated tool to systematically compare institutional protocols with evidence-based clinical practice guidelines. We analyzed alignment between the SOAP/ASRA 2024 Consensus report on postdural puncture headache (PDPH)1 and institutional protocols, flagging omissions and inconsistencies at scale.

Methods

We examined 48 of the PDPH recommendations with 5 institutional protocols uploaded on C8 Health (data from Dec 2025). We built an LLM-based AI model with a 4-step pipeline resulting in adjudication of items as “aligned”, “inconsistent” or “missing” (see Figure 1). We also tested associations between alignment and grade of evidence (A/B/C/D/I), using χ2 tests, and evaluated which recommendations are mostly aligned or missing, across institutions.

Results

After performing 240 comparisons (48 PDPH recommendations x 5 institutional protocols), our tool labeled 49 as aligned (20.42%), 12 as inconsistent (5.0%), and 180 as missing (74.58%). Alignment varied widely by site (Table 1), with Hospital 5 at 20.78% vs Hospital 3 and 4 at 14.58%. Compared with anesthesiologists' adjudication (gold standard), the system achieved 97% recall and precision with an accuracy of 96% across two test sets (96 item–site pairs). Alignment correlated strongly with evidence level.

Systematically aligned recommendations included: hydration, multimodal analgesia, epidural blood patch sterility, and follow-up until resolution.
Systematically missing recommendations included: procedural characteristics, avoidance of unsupported therapies, and primary care physician handoff.

Conclusions

Our AI-assisted guideline audit effectively surfaces misalignments in institutional protocols, achieving 96% accuracy compared with expert review, demonstrating the feasibility of an automated framework for assessing protocol alignment and relevance that can be broadly applied across the literature. Our model also demonstrates the ability to identify areas addressed and omitted systematically by institutions, emphasizing clinical needs and relevance for future statements and publications.

This approach enables rapid, scalable identification of practice gaps that may impact patient safety and quality of care, facilitating timely protocol updates.