Abstract

A key challenge in training neural networks for a given medical imaging task is often the difficulty of obtaining a sufficient number of manually labeled examples. In contrast, textual imaging reports, which are often readily available in the medical record, contain rich, but unstructured, interpretations written by experts as part of standard clinical practice. We propose to use these textual reports as a form of weak supervision to improve the image-interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune, in a transfer learning setting, for a supervised task using a small labeled dataset. The end result is a neural network that automatically interprets the imagery, without requiring textual reports during inference. 
This approach can be applied to any task where text-image pairs are readily available. We evaluated our method on three classification tasks and found consistent performance improvements, reducing the need for labeled data by 70%--98%.

Figure 1. ​The TIMNet architecture. 1) Weakly-supervised image feature learning through a text-image matching network (solid black line). 2) Downstream application training using a small dataset (dashed blue line).

Recommended citation

@article{liang2020weakly,  
  title={​Weakly-Supervised Feature Learning via ​Text and Image Matching},  
  author={Liang, Gongbo and Greenwell, Connor and Zhang, Yu and Wang, Kavuluru​, Ramakanth and Xiaoqin and Jacobs, Nathan},  
  journal={arXiv preprint arXiv:2010.03060},  
  year={2020}
}