Background: Experts manually classify positron emission tomography (PET) scans in large multi-site databases. PET is particularly challenging because the radiotracer used, the most crucial information, is unknown by the scanner; it must be entered manually at scan time, risking labeling errors. DICOM metadata can also be removed or incorrect.

Objective: To develop a deep learning PET classifier that predicts the radiotracer directly from image appearance, independent of DICOM metadata.

Conclusions: 1. The model learned meaningful tracer-specific image. features and lower performance was explained by tracer similarity, class imbalance, and cohort variation (e.g. ADNI has more older data from older scanners). 2. Accurate and generalizable deep learning models can be trained to identify PET tracer type from image appearance alone, without relying on DICOM metadata.