Background: Alzheimer's disease and related dementias (ADRD) is a leading cause of mortality and disability among the elderly over 65. Early detection is crucial for timely intervention, caregiver planning, and patient enrollment in clinical trials. Electronic health records (EHRs) present a readily accessible data source for early detection. However, existing predictive models often suffer from extreme sparsity and noise in EHRs. Recently, large language models (LLMs)--encoding broad medical knowledge, demonstrating strong clinical reasoning–have shown promising capabilities to capture weak but critical signals in EHRs for improving predictive performance.

Methods: We developed a novel pipeline that integrates supervised learning methods (SLs) with LLMs through a confidence-driven decision-making mechanism. This approach employs SLs for cases with high certainty and LLMs for more complex cases. We evaluated this collaborative approach using a large, real-world EHR dataset from Oregon Health & Science University (OHSU), consisting of over 2.5 million patient records spanning more than 20 million clinical encounters. Our methodology also incorporates in-context learning (ICL) with denoising strategies to refine LLM predictions for better performance.

Results: The integration of SLs and LLMs in our pipeline resulted in a substantial improvement in the accuracy of ADRD risk prediction, surpassing traditional SL methods and LLM-alone approaches by at least 5.2%. Implementing denoising strategies within the ICL significantly enhanced LLM’s performance, leading to an almost 30% gain. Furthermore, our analysis involving various LLaMA2-based fine-tuned medical models challenges the prevailing LLM scaling laws, demonstrating that smaller models can outperform larger ones under specific circumstances.

Conclusions: This research highlights the effectiveness of a hybrid SL-LLM approach in utilizing EHRs for the predictive diagnosis of ADRD, demonstrating potential significant impacts on patient management and care strategies. Additionally, we found that denoising markedly boosts LLM performance in our pipeline, increasing both effectiveness and efficiency. Our results also challenge conventional scaling laws for LLMs, suggesting the need for more tailored model applications in medical diagnostics.