Abstract

When training deep neural networks for medical image classification, obtaining a sufficient number of manually annotated images is often a significant challenge. We propose to use textual imaging reports, which are routinely written by clinicians, to help overcome this problem. Such textual reports are widely available in the medical record system and contain rich information about images. However, they are hard to be used as labels for neural network training. We propose to use such reports as weak supervisions for image feature representation learning. The key idea is to learn image features through an image and text matching task. The learned image feature extractor is then fine-tuned, in a transfer learning setting, for a supervised classification task using a small labeled dataset. This approach is widely applicable and useful in applications where manually annotated training images are limited, but text-image pairs are readily available. No textual reports are needed during inference. The proposed method maximizes the uses of existing data while reducing the need for labeled data significantly. We evaluate the proposed method on three classification tasks and find consistent performance improvements. The proposed method reduces the need for labeled data by 70%--98%.

Figure 1. ​The TIMNet architecture. 1) Weakly-supervised image feature learning through a text-image matching network (solid black line). 2) Downstream application training using a small dataset (dashed blue line).

Recommended citation

@article{liang2020weakly,  
  title={​Weakly-Supervised Feature Learning via ​Text and Image Matching},  
  author={Liang, Gongbo and Greenwell, Connor and Zhang, Yu and Wang, Kavuluru​, Ramakanth and Xiaoqin and Jacobs, Nathan},  
  journal={arXiv preprint arXiv:2012.XXX},  
  year={2020}
}