A New Era in Computational Pathology: A Survey on
Foundation and Vision-Language Models

PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding

---

: Dai et al.
: PA-LLaVA
: VLM/Vision Instruction Tuning

---

Entity	Links
Arxiv	Link
Code	Link
Model Weight	Link
Citation Graph	Link

Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology

---

: Zimmermann et al.
: Virchow2 and Virchow2G
: VFM

---

Entity	Links
Arxiv	Link
Model Weight	Link
Citation Graph	Link

PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology

---

: Wu et al.
: PathInsight
: VLM/Vision Instruction Tuning

---

Entity	Links
Arxiv	Link
Citation Graph	Link

When Multiple Instance Learning Meets Foundation Models: Advancing Histological Whole Slide Image Analysis

---

: Lu et al.
: Benchmarking VFMs and VLFMs

---

Entity	Links
Pre-Print	Link
Code	Link

Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation

---

: Ma et al.
: GPFM
: Combines Knowledge of VFMs and VLFMs

---

Entity	Links
Arxiv	Link
Code	Link
Model Weight	Link
Citation Graph	Link

Benchmarking Histopathology Foundation Models for Ovarian Cancer Bevacizumab Treatment Response Prediction from Whole Slide Images

---

: Mallya et al.
: Benchmarking FMs for Ovarian Cancer Bevacizumab Treatment Response Prediction

---

Entity	Links
Arxiv	Link
Citation Graph	Link

Preprocessing Pathology Reports for Vision-Language Model Development

---

: Lucassen et al.
: VLM Related Models

---

Entity	Links
MICCAI 2024 Workshop(OpenReview.net)	Link
Code	Link

Cost-effective Instruction Learning for Pathology Vision and Language Analysis

---

: Chen et al.
: CLOVER
: VLM/Visiual Instruction Tuning

---

Entity	Links
Arxiv	Link
Code	Link
Citation Graph	Link

PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains

---

: Hua et al.
: PathoDuet
: VFM

---

Entity	Links
Elsevier: Medical Image Analysis	Link
Arxiv	Link
Code	Link
Model Weight	Link
Citation Graph	Link

PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning

---

: Zhou et al.
: PathM3
: VLM

---

Entity	Links
Arxiv	Link
Citation Graph	Link

A Foundation Model for Clinical-Grade Computational Pathology and Rare Cancers Detection

---

: Vorontsov et al.
: Virchow
: VFM

---

Entity	Links
Nature Medicine	Link
Supplementary Material	Link
Arxiv	Link
Model Weight	Link
Citation Graph	Link

PathoTune: Adapting Visual Foundation Model to Pathological Specialists

---

: Lu et al.
: PathoTune
: Multi-modal prompt tuning framework to adapt a FM for pathology

---

Entity	Links
MICCAI 2024	---
Arxiv	Link
Code	Link
Citation Graph	Link

Pathology-knowledge Enhanced Multi-instance Prompt Learning for Few-shot Whole Slide Image Classification

---

: Qu et al.
: PEMP
: VLM

---

Entity	Links
ECCV 2024	---
Arxiv	Link
Citation Graph	Link

A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models

---

: Campanella et al.
: Clinical benchmark of VFMs

---

Entity	Links
Arxiv	Link
Code	Link
Citation Graph	Link

H-optimus-0

---

: Saillard et al.
: VFM

---

Entity	Links
Code	Link

Benchmarking PathCLIP for Pathology Image Analysis

---

: Zheng et al.
: Benchmarks PathCLIP

---

Entity	Links
Springer: Journal of Imaging Informatics in Medicine	Link
Arxiv	Link
Citation Graph	Link

WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering

---

: Chen et al.
: W2T
: VLM

---

Entity	Links
ECCV 2024	---
Arxiv	Link
Code	Link
Citation Graph	Link

WSI-VQA

---

: Chen et al.
: WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering
: WSI and Q/A pairs
: 977 WSIs 8,672 question-answer pairs (slide-level) with average 8.9 Q/A pairs per WSI
: The dataset is publicaly available

---

Entity	Links
Associated Paper (Arxiv)	Link
Code	Link
Training Set	Link
Validation Set	Link
Test Set	Link
Model Checkpoint	Link
Citation Graph	Link

PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration

---

: Sun et al.
: PathGen-CLIP, PathGen-LLaVA
: VLM/Vision instruction tuning

---

Entity	Links
Arxiv	Link
Code	Link
Dataset	Link
Model Weight (PathGen-CLIP)	Link
Citation Graph	Link

PathGen-1.6M

---

: Sun et al.
: PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration
: Image-Caption Pair
: 1.6 million image-caption pairs
: The dataset is publicaly available

Entity	Links
Associated paepr (Arxiv)	Link
Code	Link
Dataset	Link
Citation Graph	Link

PathAlign: A vision-language model for whole slide images in histopathology

---

: Ahmed et al.
: PathAlign
: VLM

---

Entity	Links
MICCAI Workshop (OpenReview.Net)	Link
Arxiv	Link
Citation Graph	Link

WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images

---

: Chen et al.
: MI-Gen
: VLM

---

Entity	Links
MICCAI 2024	---
Arxiv	Link
Code	Link
Citation Graph	Link

Prompting Whole Slide Image Based Genetic Biomarker Prediction

---

: Zhang et al.
: PromptBio
: VLM

---

Entity	Links
Arxiv	Link
Code	Link
Citation Graph	Link

Prompting Vision Foundation Models for Pathology Image Analysis

---

: Yin et al.
: QAP
: Prompting method for VFMs

---

Entity	Links
CVPR 2024	Link
Supplementary Material	Link
Code	Link
Presentation Video	Link
Citation Graph	Link

ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification

---

: Shi et al.
: ViLa-MIL
: VLM

---

Entity	Links
CVPR 2024	Link
Supplementary Material	Link
Code	Link

Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction

---

: Li et al.
: FiVE
: VLM

---

Entity	Links
CVPR 2024	Link
Supplementary Material	Link
Arxiv	Link
Code	Link
Citation Graph	Link

CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment

---

: Javed et al.
: CPLIP
: VLM

---

Entity	Links
CVPR 2024	Link
Supplementary Material	Link
Arxiv	Link
Code	Link
Project Page	Link
Model Weight	Link
Citation Graph	Link

A Multimodal Generative AI Copilot for Human Pathology

---

: Lu et al.
: PathChat
: VLFM

---

Entity	Links
Nature	Link
Supplementary Material	Link
Arxiv	Link
Code	Link
Video Demo	Link
Dataset: PathQABench-Public	Link
Citation Graph	Link

RudolfV: A Foundation Model by Pathologists for Pathologists

---

: Dippel et al.
: RudolfV
: VFM

---

Entity	Links
Arxiv	Link
Citation Graph	Link

Hibou: A Family of Foundational Vision Transformers for Pathology

---

: Nechaev et al.
: Hibou
: VFM

---

Entity	Links
Arxiv	Link
Code	Link
Model Weight (Hibou-L)	Link
Citation Graph	Link

Generating clinical-grade pathology reports from gigapixel whole slide images with HistoGPT

---

: Tran et al.
: HistoGPT
: VLM

---

Entity	Links
medRxiv	Link
Code	Link
Model Weight	Link
Notebook Demo	Link
Citation Graph	Link

HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction

---

: Guo et al.
: HistGen
: VLM

---

Entity	Links
MICCAI 2024	---
Arxiv	Link
Code	Link
Curated Dataset	Link
Citation Graph	Link

Automatic Report Generation for Histopathology Images Using Pre-Trained Vision Transformers and BERT

---

: Sengupta et al.
: HistoCapBERT
: VLM

---

Entity	Links
IEEE International Symposium on Biomedical Imaging	Link
Arxiv	Link
Code	Link
Model Weight	Link
Citation Graph	Link

A Whole-Slide Foundation Model for Digital Pathology from Real-World Data

---

: Xu et al.
: Prov-GigaPath
: VLFM

---

Entity	Links
Nature	Link
Supplementary Material	Link
Code	Link
Notebook Demo	Link
Model Weight	Link
Citation Graph	Link

Prov-Path

---

: Xu et al.
: A Whole-Slide Foundation Model for Digital Pathology from Real-World Data
: WSI-reports pairs
: 17, 383 WSI-reports pairs
: The entire pre-training dataset is not publicaly available. A de-identified test subset from Providence Health System by reasonable request and subject to local and national ethical approval.

---

Entity	Links
Associated Paper	Link
De-identified Subset 1	Link
De-identified Subset 2	Link
Citation Graph	Link

PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology

---

: Shaikovski et al. et al.
: PRISM
: VLFM

---

Entity	Links
Arxiv	Link
Model Weight	Link
Citation Graph	Link

A Foundation Model for Generalizable Cancer Diagnosis and Survival Prediction from Histopathological Images

---

: Yang et al.
: BEPH
: VFM

---

Entity	Links
bioRxiv	Link
Code	Link
Model Weight	Link
Citation Graph	Link

PLUTO: Pathology-Universal Transformer

---

: Juyal et al.
: PLUTO
: VFM

---

Entity	Links
Arxiv	Link
Citation Graph	Link

Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging

---

: Thota et al.
: ---
: Investigates the effect of PGD adversarial perturbation attack on PLIP

---

Entity	Links
Arxiv	Link
Code	Link
Citation Graph	Link

Impact of Layer Selection in Histopathology Foundation Models on Downstream Task Performance

---

: Aswolinskiy et al.
: Investigates the impact of layer selection for 5 VFMs

---

Entity	Links
OpenReview.net (MIDL 2024 Short Papers)	Link

Knowledge-enhanced Visual-Language Pretraining for Computational Pathology

---

: Zhou et al.
: KEP
: VFM

---

Entity	Links
ECCV 2024	Link
Arxiv	Link
Code	Link
Citation Graph	Link

Histopathology language-image representation learning for fine-grained digital pathology cross-modal retrieval

---

: Hu et al.
: ---
: VLM

---

Entity	Links
Elsevier: Medical Image Analysis	Link
Code	Link
Processed Dataset	Link
Citation Graph	Link

PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

---

: Sun et al.
: PathAsst, PathCLIP
: VLFM

---

Entity	Links
AAAI Conference on Artificial Intelligence	Link
Arxiv	Link
Code	Link
Model Weight	Link
Citation Graph	Link

Towards Large-Scale Training of Pathology Foundation Models

---

: Aben et al.
: eva
: Benchmarking of VFMs and a framework called eva

---

Entity	Links
Arxiv	Link
MIDL 2024 (eva)	Link
Code	Link
Code for eva	Link
Documentation for eva	Link
Citation Graph	Link

Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning

---

: Watawana et al.
: HLSS
: VLM

---

Entity	Links
MICCAI 2024	---
Arxiv	Link
Code	Link
Citation Graph	Link

Towards a General-Purpose Foundation Model for Computational Pathology

---

: Chen et al.
: UNI
: VFM

---

Entity	Links
Nature Medicine	Link
Supplementary Material	Link
Code	Link
Model Weight	Link
Citation Graph	Link

PathMMU

---

: Sun et al.
: PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology
: Image and Q/A Pairs
: 24,067 pathology images 33,428 Q/A
: The dataset is publicaly available

---

Entity	Links
Associated Paper (Arxiv)	Link
Code	Link
Dataset	Link
Citation Graph	Link

A Visual-Language Foundation Model for Computational Pathology

---

: Lu et al.
: CONCH
: VLFM

---

Entity	Links
Nature Medicine	Link
Supplementary Material	Link
Arxiv	Link
Code	Link
Model Weight	Link
Citation Graph	Link

Dataset from CONCH

---

: Lu et al.
: A Visual-Language Foundation Model for Computational Pathology
: Image-Caption Pair
: 1,786,362 image-caption pairs
: The dataset contains two subsets PMC-Path (data from PubMed) and EDU (data extracted from educational notes). The unprocessed PubMed data is publicaly available. Educational resources are not publicaly available as they are are subject to copyright terms of publishers.

---

Entity	Links
Associated Paper	Link
Code	Link
Unprocessed PubMed Data	Link
Citation Graph	Link

In-context learning enables multimodal large language models to classify cancer pathology images

---

: Ferber et al.
: In-context learning through prompting strategy

---

Entity	Links
Arxiv	Link
Citation Graph	Link

Foundation Models for Histopathology Fanfare or Flair

---

: Alfasly et al.
: Benchmarking foundation models

---

Entity	Links
Elsevier: Mayo Clinic Proceedings: Digital Health	Link

Low-Resource Fine-Tuning of Foundation Models Beats State-of-the-Art in Histopathology

---

: Roth et al.
: Low-resource training of FMs

---

Entity	Links
IEEE International Symposium on Biomedical Imaging	Link
Arxiv	Link
Citation Graph	Link

PathLDM: Text Conditioned Latent Diffusion Model for Histopathology

---

: Yellapragada et al.
: PathLDM
: Text-to-image generation by utilizing a diffusion model

---

Entity	Links
WACV	Link
Supplementary Material	Link
Arxiv	Link
Code	Link
Citation Graph	Link

The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification

---

: Qu et al.
: FSWC
: VLM

---

Entity	Links
NeurIPS 2023	Link
Supplementary Material	Link
Arxiv	Link
Code	Link
Citation Graph	Link

Automatic Report Generation for Histopathology images using pre-trained Vision Transformers

---

: Sengupta et al.
: HistoCap
: VLM

---

Entity	Links
Machine Learning for Health (ML4H) Symposium 2023	---
Arxiv	Link
Code	Link

What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioning

---

: Qin et al.
: SGMT
: VLM

---

Entity	Links
Arxiv	Link
Citation Graph	Link

Domain specific optimization and diverse evaluation of self-supervised models for histopathology

---

: Lai et al.
: Benchmarking SSL schemes for FMs

---

Entity	Links
Google Research	Link
Arxiv	Link
Citation Graph	Link

Computational Pathology at Health System Scale – Self-Supervised Foundation Models from Billions of Images

---

: Campanella et al.
: 3B-CPath
: VFM

---

Entity	Links
OpenReview.net (AAAI 2024 SSS on Clinical FMs)	Link
Arxiv	Link
Code	Link
Citation Graph	Link

CLIPath: Fine-Tune CLIP with Visual Feature Fusion for Pathology Image Analysis Towards Minimizing Data Collection Efforts

---

: Lai et al.
: CLIPath
: VLM

---

Entity	Links
ICCV 2023	Link
Citation Graph	Link

Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling

---

: Filiot et al.
: Phikon
: VFM

---

Entity	Links
medRxiv	Link
Code	Link
Model Weight	Link
Raw Data	Link
Pre-Processed Data	Link
Citation Graph	Link

A Visual–Language Foundation Model for Pathology Image Analysis Using Medical Twitter

---

: Huang et al.
: PLIP
: VLFM

---

Entity	Links
Nature Medicine	Link
Supplementary Material	Link
Code	Link
Model Weight	Link
Dataset (OpenPath)	Link
Citation Graph	Link

OpenPath

---

: Huang et al.
: A Visual–Language Foundation Model for Pathology Image Analysis Using Medical Twitter
: Image-Caption Pair
: 208,414 image-caption pairs
: The dataset is publicaly available

---

Entity	Links
Associated Paper	Link
Dataset (OpenPath)	Link
Citation Graph	Link

Caption generation from histopathology whole-slide images using pre-trained transformers

---

: Guevara et al.
: VLM

---

Entity	Links
Medical Imaging with Deep Learning 2023	Link

Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images

---

: Lu et al.
: MI-Zero
: VLM

---

Entity	Links
CVPR 2023	Link
Supplementary Material	Link
Arxiv	Link
Presentation Video	Link
Code	Link
Model Checkpoint	Link
Citation Graph	Link

Benchmarking Self-Supervised Learning on Diverse Pathology Datasets

---

: Kang et al.
: Lunit
: VFM

---

Entity	Links
CVPR 2023	Link
Supplementary Material	Link
Arxiv	Link
Project Page	Link
Code	Link
Model Weight	Link
Citation Graph	Link

Enhanced descriptive captioning model for histopathological patches

---

: Elbedwehy et al.
: VLM

---

Entity	Links
Springer: Multimedia Tools and Applications	Link
Citation Graph	Link

Vision-Language Transformer for Interpretable Pathology Visual Question Answering

---

: Naseem et al.
: TraP-VQA
: VLM

---

Entity	Links
IEEE Journal of Biomedical and Health Informatics	Link
Citation Graph	Link

Transformer-based unsupervised contrastive learning for histopathological image classification

---

: Wang et al.
: CTransPath
: VFM

---

Entity	Links
Elsevier: Medical Image Analysis	Link
MICCAI/Springer Version	Link
Code	Link
Model Weight	Link
Citation Graph	Link

Inference of captions from histopathological patches

---

: Tsuneki et al.
: ---
: VLM

---

Entity	Links
PMLR	Link
Code	Link
Dataset	Link
Citation Graph	Link

ARCH

---

: Gamper et al.
: Multiple instance captioning: Learning representations from histopathology textbooks and articles
: Image-caption pairs
:11,816 image-caption pairs
: The dataset is publicaly available.

---

Entity	Links
Associated paper (CVPR 2021)	Link
Supplementary Material	Link
Dataset	Link
Citation Graph	Link

PathVQA

---

: He et al.
: PathVQA: 30000+ Questions for Medical Visual Question Answering
: Image-caption pairs
: 4,998 images 32,799 Q/A pairs
: The dataset is publicaly available.

---

Entity	Links
Associated paper (Arxiv)	Link
Dataset	Link
Citation Graph	Link

Evaluating and interpreting caption prediction for histopathology images

---

: Zhang et al.
: PathCap
: VLM

---

Entity	Links
PMLR	Link
Code	Link
Citation Graph	Link