Maria Majeed1 Anna France1 Darren Reynolds1 Jia Min Lim2 Love Goyal13 Shermaine Pan4

Affiliations:

1Department of Proton Beam Therapy, The Christie NHS Foundation Trust, Manchester, UK

2Wythenshawe Hospital, Manchester, Foundation Trust

3Division of Cancer Sciences, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester, UK

4The Christie NHS Foundation Trust, Manchester UK

Introduction

Real-world data (RWD) in proton beam therapy provide clinical insights of an unbiased cohort of patients on their survival outcomes and toxicities following treatment. However, these datasets present significant analytical challenges, including incomplete baseline and follow-up information, and limited statistical precision due to small heterogenous cohorts. These limitations may affect the reliability of outcomes estimates and highlight the importance of methodological rigour in the design, analysis and interpretation of RWD studies.

Survival Inference:

In a cohort of patients with non-germinomatous germ cell tumours (NGGCT), a subgroup comparison of progression free survival (PFS) [77.8%(54.9%, 100%) in high-risk and 96.3%(89.4%, 100%) in standard-risk] showed a trend towards significance p=0.068 (high-risk N=9, event=2; standard-risk N=28, event=1). Even if the p-value had been statistically significant, the result would still have high degree of variability and uncertainty due to small numbers. Also, there were twice as many events in the smaller group compared to the larger group. 

Therefore, a smaller risk set results in each event having higher impact on Kaplan-Meier (KM) estimates, as KM estimate is essentially dependent on number of events over number still at risk prior to time of event. Alternatively, when localised and metastatic groups were compared and despite that the number of patients at risk were comparable in both groups [22 vs 15, events 2 vs 1], estimates had higher uncertainty with CI overlapping. So, statistical significance alone may not indicate reliability and should be interpreted along with clinical context.

Accounting for outcome distribution:

In a cohort of 23 patients with salivary gland tumours, oral cavity mean dose was assessed across three grades of mucositis: absent 28%(7), grade 1 52%(13), and grade 2/3 20%(5). A significant difference was noted only between patients with no mucositis and grade 2/3 mucositis using Dunn’s test for pairwise comparisons. Including all grades, a moderate positive Spearman correlation was noted (r=0.56, p=0.0058,n=23). Based on area under the ROC analysis (AUC), the predictive performance of mean oral cavity dose with any grade of mucositis in ROC analysis indicated a good discriminative ability with AUC of 0.83 and ROC value of 0.76 (95% CI: 0.51,1.00). Ordinal logistic regression confirmed this association β = 0.13 per Gy; (CI:0.028-0.224, p = 0.011).

Using Youden's index, the optimal cut-off for oral cavity mean dose to predict the likelihood of mucositis was 5 Gy (Youden J = 0.67), with 81% sensitivity and 86% specificity (AUC = 0.84). The threshold for 5 Gy was then used in Firth’s penalised logistic regression that showed mean oral cavity dose ≥5 Gy had 19.2 fold higher odds of developing any grade mucositis compared with those receiving <5 Gy (OR = 19.19, 95% CI: 2.30–159.99, p = 0.006). There were 15/16 cases of mucositis in the group having ≥5 Gy and only 3/9 cases of mucositis in those having <5 Gy mean oral cavity dose.

This resulted in quasi-separation indicating almost all or most of the event occurring in the group receiving ≥5 Gy mean oral cavity dose. Even though, a statistically significant association was noted in logistic regression and ROC analysis, the results were not statistically reliable and meaningful with wide CIs and partial separation of outcomes in the two categories (<5Gy versus ≥5 Gy). This highlights the sensitivity of statistical inference by unstable estimates particularly in real-world data susceptible to separation and smaller sample size.

Conclusion

These findings indicate that even small heterogeneous cohorts can generate clinically meaningful insights. Although statistical uncertainty remains due to small sample sizes and heterogeneity, careful methodological handling and rigorous analytical approaches can preserve the credibility and interpretation of the results. These cohorts represent a larger proportion of the population of interest, patients receiving treatment and reflecting outcomes relevant to the real-world setting. Despite methodological and analytical limitations, RWD about rare tumours provide valuable context about clinical decision-making and highlight the importance of methodological rigour and valid interpretation.