Ahmad Elallaf1, Yu Zhang2, Yuktha Priya Masupalli1, Jeong Yang1, Young Lee1, Zechun Cao1, Gongbo Liang1
1Texas A&M University-San Antonio, 2Boise State University
The Winter Conference on Applications of Computer Vision (WACV) Workshops, 2026
Figure 1: Illustration of the inherent many-to-many relationships in cross-modal datasets. Although MS-COCO annotates only a single caption as the positive match to one image (blue arrows), human raters often identify multiple additional plausible matches (pink dashed arrows). Such unannotated positives create false negatives that violate the one-to-one assumption commonly enforced in contrastive learning, motivating models capable of handling ambiguity and uncertainty in image–text alignment.
Vision-language foundation models have emerged as powerful general-purpose representation learners with strong potentials for multimodal understanding, but their deterministic embeddings often fail to provide the reliability required for high-stakes biomedical applications. This work introduces MedProbCLIP, a probabilistic vision–language learning framework for chest X-ray and radiology report representation learning and bidirectional retrieval. The proposed method models image and text representations as Gaussian embeddings through a probabilistic contrastive objective that explicitly captures uncertainty and many-to-many correspondences between radiographs and clinical narratives. A variational information bottleneck mitigates overconfident predictions, while MedProbCLIP employs multi-view radiograph encoding and multi-section report encoding during training to provide fine-grained supervision for clinically aligned correspondence, yet requires only a single radiograph and a single report at inference.
Although datasets typically assign one caption to one image, real image–text relationships are rarely one-to-one. As shown in Figure 1, human raters often identify a single caption may accurately describe many visually similar images (or vice versa). These unannotated positives become false negatives, misleading standard contrastive learning models that assume a strict one-to-one alignment.
This structural ambiguity exists even more strongly in medical imaging-reports summarize multiple views, findings overlap across studies, and similar pathologies appear in many radiographs, resulting a fundamental mismatch:
MedProbCLIP is motivated by addressing this mismatch requires models that can represent uncertainty, model multiple plausible matches, and avoid overconfident errors.
Figure 2 illustrates the architecture of MedProbCLIP. The model receives two image inputs (e.g., PA and lateral views) and two text inputs (e.g., Findings and Impression) for each study. Each deterministic feature is passed through a Distribution Learning Model (i.e., probabilistic projector) to obtain a Gaussian embedding parameterized by mean and variance. During training, all image-text and within-modality pairs are compared using contrastive stochastic distance and optimized with probabilistic negative log-likelihood losses, combined with KL-based variational regularization. This multi-view, probabilistic formulation enables MedProbCLIP to learn uncertainty-aware and semantically calibrated cross-modal embeddings for medical retrieval.
Figure 2: Overview of the MedProbCLIP architecture.
We evaluated MedProbCLIP on the MIMIC-CXR dataset, benchmarking its performance against CLIP, PCME++, and CXR-CLIP across three key dimensions: cross-modal retrieval, zero-shot classification, and robustness. As shown in Tables 1 and 2, MedProbCLIP consistently outperformed baseline models in retrieval and classification tasks. To further assess reliability, Figure 3 illustrates Selective Retrieval Performance via Risk-Coverage curves. Our analysis focuses on two primary metrics: 1) the Area Under the Risk-Coverage Curve (AURC), where a lower value signifies superior uncertainty calibration; and 2) curve stability, where a flatter trajectory indicates the model maintains reliable predictions even when required to answer a larger share of queries. These results demonstrate MedProbCLIP's superior calibration and its potential for safer deployment in clinical workflows.
Figure 3: Selective retrieval performance demonstrates MedProbCLIP's superior calibration and potential for safer clinical deployment.
This material is based upon work supported by the National Science Foundation under Award No. 2334243. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
The computational portion of this research utilized the GPU cluster provided by the High Performance Computing Research Center at Texas A&M University-San Antonio. We gratefully acknowledge the center's support in facilitating access to these resources.
If you find this work useful in your research, please consider citing our paper:
