Purpose:

To evaluate the diagnostic accuracy of open-source foundation models on plain film X-rays, including fracture detection on musculoskeletal X-rays (MSK X-Rays) and pneumothorax on chest X-ray (CXR). 

Methods: 

A retrospective evaluation was conducted using 500 MSK X-rays and 394 CXRs obtained from Oxford University Hospitals. Suitable models were identified via a literature search, and official implementation methods were applied to evaluate zero-shot inferencing without fine tuning.

Ground truth was defined independently by two radiologists. Difficulty scores were assigned to each image. Sensitivity, specificity, and, where possible, area under receiver operating characteristic (ROC) curve values were calculated. 

Results: 

Five out of eight models were able to inference all images. Overall, for all diagnoses, sensitivity ranged from 0.0% (95% confidence interval [CI] 0.0-1.7) to 94.1% [89.7% - 97.0%]. Specificity ranged from 19.2% (14.1-25.3) to 98.2 (95.9-99.4). M05 was the only model to demonstrate a high sensitivity (78.5 [95% CI 71.9-84.2]) and specificity (95.2 [91.3-97.7]) for PTX diagnosis on CXRs. With fracture detection on MSK X-ray, no model demonstrated both sensitivity and specificity over 50%.

Conclusions:

Open-source foundation models performed poorly. Only M05 demonstrated good sensitivity and specificity on CXRs. All other models were only strong in one metric. Therefore, these models are not recommended for clinical use without further development.