Abstract

A key challenge in training neural networks for a given medical imaging task is often the difficulty of obtaining a sufficient number of manually labeled examples. In contrast, textual imaging reports, which are often readily available in the medical record, contain rich, but unstructured, interpretations written by experts as part of standard clinical practice. We propose to use these textual reports as a form of weak supervision to improve the image-interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune, in a transfer learning setting, for a supervised task using a small labeled dataset. The end result is a neural network that automatically interprets the imagery, without requiring textual reports during inference. 
This approach can be applied to any task where text-image pairs are readily available. We evaluated our method on three classification tasks and found consistent performance improvements, reducing the need for labeled data by 70%--98%.

Figure 1. ​The TIMNet architecture. 1) Weakly-supervised image feature learning through a text-image matching network (solid black line). 2) Downstream application training using a small dataset (dashed blue line).

Recommended citation

@article{liang2020weakly,  
  title={Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging}, 
  author={Liang, Gongbo and Greenwell, Connor and Zhang, Yu and Xing, Xin and Wang, Xiaoqin and Kavuluru, Ramakanth and Jacobs, Nathan},
  journal={IEEE Journal of Biomedical and Health Informatics}, 
  year={2022},
  volume={26},
  number={4},
  pages={1640-1649},
  doi={10.1109/JBHI.2021.3110805}​
}