Audio-Based Biomarkers for AD/ADRD: A Multilingual Deep Learning Approach

IGC Pharma

Background: The lack of non-invasive, accessible, and scalable biomarkers for Alzheimer’s disease and related dementias (AD/ADRD) remains a major challenge. While blood-based biomarkers are actively studied, alternative modalities show promise. Audio biomarkers, easily collected via standardized tasks or natural conversation, can capture speech features associated with AD/ADRD pathology [1-3].

Methods: We extracted multilingual audio recordings (English, Spanish, Mandarin, and German) from the DementiaBank subrepository of TalkBank. Figure 1 presents distributions of the data for training and testing, and shows the diagnosis distribution and the type of tasks available in each language. The diagnoses found in this dataset consist of healthy controls (HC), mild cognitive impairment (MCI), Alzheimer’s disease (AD), and other dementias (OD). Time-aligned transcripts were used to isolate participant speech. All recordings were processed through a voice enhancement pipeline to improve clarity and reduce acoustic variability, including background noise reduction, dynamic range compression, frequency equalization, and loudness normalization. 

We evaluated several models, including distilled and non-distilled Whisper [4] variants and a hybrid transformer architecture, for temporal modeling, with optional early/late fusion of demographic metadata and audio features. Figure 2 shows a schema of the original Whisper architecture, and the audio segmentation and feature fusion approaches. Recordings were segmented into 30-second chunks to match Whisper’s encoder requirements. However, the audios were also split into 10-second chunks for the transformer architecture. Participant-level predictions were computed by averaging segment-level class probabilities for Whisper models, while the hybrid architecture produced participant-level predictions directly. To address language imbalance and improve generalizability, we introduced a speech-to-speech translation pipeline as shown in Figure 3.

Results: The most relevant experiments are displayed in Table 1. Whisper Large achieved the best overall performance across diagnostic classes (HC, MCI, AD, and OD), particularly when intermediate layers were frozen. AUPRC and accuracy were 0.799/0.757 (HC), 0.440/0.677 (MCI), 0.798/0.728 (AD), and 0.399/0.899 (OD), respectively. Incorporating demographic metadata improved OD classification, suggesting utility for certain clinical subtypes. Our hybrid architecture effectively captured temporal speech dynamics and achieved the strongest OD performance, potentially due to longer recordings in this group. The speech-to-speech translation approach improved predictions for most non-healthy classes in Mandarin and Spanish, with notable gains for AD, as shown in Table 2.

Conclusions: In summary, our results show that larger pretrained models, careful adaptation via partial fine-tuning, and modeling speech patterns over time are key to improving the accuracy and reliability of speech-based AD/ADRD prediction. Higher capacity models showed gains from large audios which points to two viable strategies for future data collection: maximizing variability through short recordings when using small models or leveraging longer recordings to enrich sequential speech modeling. Finally, translating non-English speech, while preserving speaker characteristics, can effectively reduce the language domain gap and improve model performance.