Assessing The Utility Of Unstructured Clinical Text For Dementia Risk Prediction Using Natural Language Processing

Problem Statement

Pre-trained Language Models (e.g. BERT, Qwen, Llama) show success in risk prediction but interpreting black-box models is challenging.

Sparse Autoencoders (SAEs) combined with LLMs can be used to generate interpretations of transformer model internal mechanisms.

Generated interpretations are often highly predictive but lack grounding in domain knowledge – critical for clinical applications.

Methods

We adapted an SAE-based interpretation framework to risk prediction and extended it to check explanations against existing literature.

We evaluated model interpretations using three metrics: fidelity (alignment with model), predictivity (alignment with prediction target), and retrieval score (alignment with literature).

We applied this approach to dementia risk prediction using 10,046 notes from 4,300 patients at an urban academic memory clinic.