Domain-Specific Foundation Models for Medical Imaging

Collaborators: Rogier van der Sluijs, Akshay Chaudhari, Daniel Rubin

Advised by: Daniel Rubin, MD, MS

Professor of Biomedical Data Science, Radiology, and Medicine (Biomedical Informatics)

and (by courtesy) Computer Science and Ophthalmology,

Stanford University

Advised by: Akshay Chaudhari, PhD

Assistant Professor

Radiology and (by courtesy) Biomedical Data Science

Stanford University

Self-supervised foundation models have seen tremendous success in image recognition benchmarks with natural images without the need for labelled data. Such frameworks rely on effective “augmentations” or “transformations” of the original image to provide strong inductive bias and thus help learn representations that retain high level semantic content. Most concurrent works directly make use of transformations that have seen empirical success in natural vision tasks to other domains such as medical imaging. While models pre-trained on natural images (using either a supervised or a self-supervised strategy) have been shown to perform well on medical imaging tasks via transfer learning, the potential benefits of designing transformations specific to medical images in such foundation models remain largely unexplored. Through our research, we designed custom domain-specific transformations for medical imaging analyses and systematically study the effects of these augmentations on the learned representations and on downstream tasks. Check out our publications for more details.