Histo-VL: Evaluating Robustness and Clinical Reliability of Histopathology Vision-Language Foundation Models
Mohamed Bin Zayed University of Artificial Intelligence, Khalifa University and Shaukat Khanum Cancer Hospital
Histo-VL is a large-scale multimodal benchmark for stress-testing histopathology vision–language models (VLMs) across 26 organs, 32 cancer types, and 5.3M patches from 14 international cohorts. Organized into 7 clinically defined diagnostic tasks, it evaluates 9 VLMs under magnification shifts, stain variation, adversarial perturbations, and textual corruptions. Results reveal that top models (KEEP, CONCH) drop up to 34% in balanced accuracy under adversarial attack, are universally miscalibrated, and fall short of expert pathologists in cancer grading — exposing standard benchmark scores as an incomplete proxy for clinical readiness. Histo-VL advocates for systematic stress testing as a prerequisite before deploying histopathological VLMs in clinical settings.
The paper was accepted in MIUA 2026. Link to the paper will be provided upon publishing.
The following is a link to an earlier version preprint
- Clone this repo and the CONCH github repository.
- Create an environment using the provided .yml file
conda env create -f benchmark.yml
conda activate benchmark
cd CONCH
pip install --upgrade pip
pip install -e .
pip install ../timm_ctp.tar --no-deps
- Install requirements in the updated_requirements file.
- Add the relative paths in :
- The paths to the data, caching and results folder in the dotenv file : "plip/reproducibility/config_example.env"
- The paths to the CONCH and MI-Zero models in : "plip/reproducibility/factory.py". The CONCH models can be downloaded from here, and MI-Zero models from here
- Path to MI-Zero configuration path in : "plip/src/models/factory.py"
- The captions for both available in the BenchCaption folder as well as the captions and splits for other experiments as well.
- Run the following bash file in "plip/reproducibility" :
bash zero_shot.sh
- To run experiments for the adversarial image or text corruptions or caption ensembling, you can specify that in the zero_shot.sh bash file or in plip/reproducibility/train.py
- To create artifacts of a specific dataset, you can run the script at plip/reproducibility/generate_validation_datasets/artifactor.sh, which allows the creation of multiple types of artifacts, including blurring, rotation, addition of fat, squamous epithelial cells, and more.
| Model | Model | Paper Link |
|---|---|---|
| PLIP | Model | Paper |
| QuiltNet | Model | Paper |
| BiomedCLIP | Model | Paper |
| Mi-Zero | Model | Paper |
| CONCH | Model | Paper |
| KEEP | Model | Paper |
| PathCLIP | Model | Paper |