Abstract

Vision-language models (VLMs) are usually evaluated as one-shot image-to-text systems: the model views an image once and immediately produces an answer. This differs from clinical image interpretation, where radiologists iteratively inspect suspicious regions, adjust image display, compare appearances, consult evidence, and revise their impression before reporting. We present GAZE, a self-directed multimodal AI framework for rare brain MRI that allows a medical VLM to perform multi-turn reasoning using viewer-level tools and retrieval, rather than relying on a single forward pass.

GAZE enables the model to call tools including zoom, window-level adjustment, contrast manipulation, edge/symmetry inspection, PubMed literature retrieval, and Open-i radiology image retrieval. Outputs are schema-validated, and every tool call is recorded, creating an auditable trace of how the final answer was produced. We evaluated GAZE on NOVA, a benchmark of 906 brain MRI cases spanning 281 rare neurological conditions, under a joint protocol assessing lesion localisation, captioning, and diagnostic accuracy from the image alone, without task-specific fine-tuning.

GAZE achieved 58.2 mAP@0.3 for lesion localisation and 34.9% Top-1 diagnostic accuracy. Tool use particularly benefited rare conditions: the proportion of cases with IoU > 0.3 increased from 17% to 58% for diagnoses with three or fewer examples. However, retrieval ablations also revealed an important safety failure mode: diagnostic accuracy can improve while localisation worsens, meaning diagnosis-only evaluation may falsely suggest improvement.

These findings show that safe agentic radiology AI should be evaluated not only by what diagnosis it gives, but also by where it points and how it reached its answer. GAZE reframes medical VLM evaluation from one-shot prediction toward auditable, radiologist-like image-reading workflows.