Psychometric Equivalence of Proprietary and Non-Proprietary Neuropsychological Test Configurations in the NACC Uniform Data Set
•The National Alzheimer’s Coordinating Center (NACC) replaced proprietary neuropsychological instruments with non-proprietary alternatives with the transition from Uniform Data Set (UDS) version 2 to UDS version 3 in 2015.
•Traditional approaches to addressing battery transitions cannot assess preservation of domain-level measurement precision and clinical validity.
•We compare proprietary and non-proprietary test configurations using co-calibrated item bank parameters on two criteria: (i) measurement precision and (ii) association with clinical dementia severity in the NACC crosswalk sample.
Methods
•In 2014, NACC crosswalk study participants with a range of dementia status were administered both the UDS 2 and UDS 3 neuropsychological batteries (Table 1).
•We used previously established co-calibrated item parameters to derive harmonized scores for three cognitive domains: memory, executive functioning, and language across proprietary and non-proprietary sets of items
•The proprietary and non-proprietary items represent components that differ between UDS versions. Overlapping items were administered in both UDS1/2 and 3 batteries (Table 2).
•For each domain, we defined two configurations:
(a) Proprietary configuration: proprietary + overlapping UDS items
(b) Non-proprietary configuration: non-proprietary + overlapping UDS items
•We derived standard error of measurement (SEM) curves for each configuration across the range of each cognitive domain.
•We examined associations between domain scores and the Clinical Dementia Rating (CDR) Global score, dichotomized as no impairment (CDR = 0) versus dementia (CDR = 1-3), excluding participants with CDR = 0.5. We evaluated discriminative accuracy using areas under the ROC curve (AUC), and applying the DeLong test to compare the performance of the logistic regression models (Figure 1).
Results
•Measurement precision was plotted using SEM curves comparing proprietary and non-proprietary test configurations (Figure 2).
•The proprietary and non-proprietary configurations demonstrate similar AUC and sensitivity (Figure 1).
•Odds ratios of CDR and domain score association were not significantly different for executive functioning and language. Slight significant difference was detected for memory (p=0.048), but the effect sizes in proprietary and non-proprietary are very large and the difference between them is not clinically meaningful.
Discussion
•UDS3 non-proprietary neuropsychological instruments have psychometric properties similar to those of the proprietary instruments they replaced.
•We observed somewhat lower SEM for the proprietary memory configuration at the extreme low end of the latent trait distribution. However, this difference was confined to a region with relatively few participants and did not translate into meaningful differences in associations with clinical severity.
•The same approach can be applied to other research studies with battery changes due to licensing constraints, evolving measurement priorities or the need for cross-cultural adaptation.
•The approach can also be used to inform new studies designing batteries using non-proprietary instruments.
