New Directions for Structure Segmentation, Phenotyping, and Language Interfacing in Histopathology

Progress in deep learning brings new tools and methods that start reshaping pathology practice and advancing research in oncology. As a postdoctoral researcher in the Mahmood Lab (Harvard Medical School, 2022–), I played a pivotal role in the development and evaluation of “foundation models” for pathology – general-purpose models that can be used for various downstream tasks. Despite advances, foundation models still face several key limitations that restrict their widespread adoption. First, these models are designed to provide “non-human interpretable” data representation, providing little actionable insights to practitioners. Moreover, they lack robust language interfacing capabilities, especially in providing quantitative information. Finally, they are predominantly focused on hematoxylin & eosin (H&E) staining, leaving other histopathology modalities largely untouched. In this context, I hypothesize that foundation models for pathology will remain limited in impact unless they incorporate key additional features: (a) off-the-shelf structure segmentation and phenotyping for downstream analysis, (b) interactive language interfacing capable of answering quantitative queries, and (c) expanded support for other modalities, e.g., immunohistochemistry. To bring these ideas to life, I introduce DeepSPIM. In DeepSPIM, I will semi-automate data labeling to train segmentation models that will be combined with large language modeling to build a language interface. I will evaluate the translational capabilities of the models in assisting pathologists with routine clinical tasks, and, exploring interpretable morphological correlates of treatment response in oncology. The goals of DeepSPIM are closely aligned with the technical skills I developed throughout my PhD and postdoc. Leveraging my mentoring experience, professional network, and support of the University of Lausanne and the ERC Program, I aim to establish an independent research group in computational pathology.