BRIDGE (Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text)

📜 Background

Large Language Models (LLMs) have demonstrated transformative potential in healthcare, yet concerns remain around their reliability and clinical validity across diverse clinical tasks, specialties, and languages. To support timely and trustworthy evaluation, building upon our systematic review of global clinical text resources, we introduce BRIDGE, a multilingual benchmark that comprises 87 real-world clinical text tasks spanning nine languages and more than one million samples. The first leaderboard release systematically evaluated 52 state-of-the-art LLMs; the leaderboard has since been expanded through regular updates.

Key Features: Real-world Clinical Text, 9 Languages, 9 Task types, 14 Clinical specialties, 7 Clinical document types, 20 Clinical applications covering 6 clinical stages of patient care.

More Details can be found in our BRIDGE paper and systematic review, and the comprehensive leaderboard is available at BRIDGE Leaderboard.

This project is led and maintained by the team of Prof. Jie Yang and Prof. Kueiyu Joshua Lin at Harvard Medical School and Brigham and Women's Hospital.

📢 Updates

• 🗓️ 2026/07/29: Unified local/API inference artifacts, added optional vLLM logits and label-probability capture, exact post-truncation input and few-shot ID recording, append-only resumable checkpoints, reproducibility metadata, and refreshed inference/evaluation notebooks.
• 🗓️ 2026/04/08: We continue to evaluate the latest models. BRIDGE Leaderboard now includes 107 evaluated models.
• 🗓️ 2025/06/03: Updated the leaderboard with 21 additional models, for 75 evaluated models in total.
• 🗓️ 2025/04/28: BRIDGE Leaderboard V1.0.0 is now live with 54 evaluated models.
• 🗓️ 2025/04/28: Our paper BRIDGE is available on arXiv.

🛠️ How to Use?

1. Download the BRIDGE Dataset

All fully open-access datasets in BRIDGE are available in BRIDGE-Open. To ensure leaderboard fairness, we publicly release five completed few-shot examples for each task and all testing samples with instruction/input fields. Regulated-access clinical datasets cannot be directly published, but task descriptions and original data sources are listed in the BRIDGE paper.

Put task files under dataset_raw/:

dataset_raw/<task_name>.SFT.json
dataset_raw/example/<task_name>.example.json
dataset_raw/example_cot/<task_name>.example.json

2. Prepare Local Configs

Real API keys and local checkpoint paths are intentionally not committed. Create local files from the templates:

cp configs/BRIDGE.example.yaml configs/BRIDGE.yaml
cp configs/API.key.example.yaml configs/API.key.yaml
cp configs/dict_model_path.example.json configs/dict_model_path.json

configs/dict_model_path.json maps model names to local checkpoint paths. configs/API.key.yaml stores provider credentials, including the AZURE-OPENAI block used by the Azure example notebook.

3. Run Local Inference

The recommended local runtime is vLLM:

mamba activate vllm
python main.py \
  --model_name gpt-oss-20b \
  --gpus 0,1 \
  --engine vllm \
  --config configs/BRIDGE.yaml \
  --model_file configs/dict_model_path.json

Configure tasks, data paths, result paths, prompt modes, decoding modes, and token budgets in configs/BRIDGE.yaml. Supported prompt modes are direct, cot, direct-N-shot, and cot-N-shot.

Results are saved as:

result/<task_name>/<model_name>/<task_name>-<prompt_mode>-<decoding>-<seed>.result.json
result/<task_name>/<model_name>/<task_name>-<prompt_mode>-<decoding>-<seed>.result.config.yaml
result/<task_name>/<model_name>/<task_name>-<prompt_mode>-<decoding>-<seed>.result.metadata.json

The result row stores the input actually sent to inference: local runs keep the post-truncation patient text in input, while API runs keep the exact formatted messages. example records only the few-shot IDs that actually fit in the context window. The adjacent config snapshot and metadata manifest capture the effective settings, file hashes, Git state, and runtime versions.

For long vLLM runs, save_every_n_batches appends new rows to *.result.partial.jsonl. Set resume_partial_results: true to resume an interrupted exact prefix; the checkpoint is removed after the final result is written successfully.

To capture vLLM generation traces or task label probabilities, keep the regular run config and add the optional logits config:

python main.py \
  --model_name gpt-oss-20b \
  --config configs/BRIDGE.yaml \
  --logits_config configs/logits.example.yaml

Without --logits_config, logits capture is disabled.

4. Evaluate Results

Evaluation is configured separately from inference. Built-in BRIDGE tasks use evaluation/evaluate.BRIDGE.yaml:

python -m evaluation.bridge
python -m evaluation.bridge --config evaluation/evaluate.BRIDGE.yaml

Project-specific tasks use the separate task entry point:

python -m evaluation.task --config evaluation/evaluate.task.yaml

Set models, prompt modes, task types, and task subsets in the YAML file. The evaluation code warns when a result file does not align with the expected task rows, which supports partial sampling runs while keeping data-integrity issues visible.

5. Run Commercial API Inference

Commercial API inference uses the unified run_API.sh entry point. Models, tasks, provider parameters, and optional automatic evaluation are read from YAML:

bash run_API.sh
bash run_API.sh --provider azure --config configs/API.provider.azure.yaml --model gpt-4o --foreground
bash run_API.sh --dry-run

Azure OpenAI, Claude, and Gemini write the same result JSON format as local inference, so the evaluation workflow is shared. Request/response traces are stored under each model result directory in api_trace/ unless disabled in YAML. API config snapshots are automatically redacted. See configs/API_USAGE.md for the detailed API workflow.

6. More Documentation

• USAGE.md: detailed runtime, data, inference, and evaluation guide.
• configs/API_USAGE.md: commercial API inference, trace, batch, and evaluation guide.
• configs/README.result.md: ordinary local/API result JSON format.
• configs/README.result.logits.md: logits/probability sidecar result format.
• configs/README.md: runtime config templates and local-only config paths.
• configs/README.logits.md: logits YAML parameter reference.
• scripts/README.md: single-model, multi-model, and delayed launch scripts.
• notebooks/README.md: interactive notebook examples.
• dataset/README.md: task data loading and prompt setup.
• model/README.md: local inference, logits, metadata, and unified API internals.
• evaluation/README.md: maintained evaluation entry point and YAML examples.
• tests/README.md: lightweight regression tests.

7. Update the Leaderboard

To submit model results to BRIDGE Leaderboard, send the generated result folder to the maintainers. We will update the leaderboard regularly and notify you when results are added.

🤝 Contributing

We welcome and greatly value contributions and collaborations from the community! If you have clinical text datasets that you would like to share for broader exploration, please contact us! We are committed to expanding BRIDGE while strictly adhering to appropriate data use agreements and ethical guidelines. Let's work together to advance the responsible application of LLMs in medicine!

🚀 Donation

BRIDGE is a non-profit, researcher-led benchmark that requires substantial resources (e.g., high-performance GPUs, a dedicated team) to sustain. To support open and impactful academic research that advances clinical care, we welcome your contributions. Please contact Prof. Jie Yang at jyang66@bwh.harvard.edu to discuss donation opportunities.

📬 Contact Information

If you have any questions about BRIDGE or the leaderboard, feel free to reach out!

• Leaderboard Managers: Jiageng Wu (jiwu7@bwh.harvard.edu), Kevin Xie (kevinxie@mit.edu), Bowen Gu (bogu@bwh.harvard.edu)
• Benchmark Managers: Jiageng Wu, Bowen Gu
• Project Lead: Jie Yang (jyang66@bwh.harvard.edu)

📚 Citation

If you find this leaderboard useful for your research and applications, please cite the following papers:

@article{BRIDGE-benchmark,
    title={BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text},
    author={Wu, Jiageng and Gu, Bowen and Zhou, Ren and Xie, Kevin and Snyder, Doug and Jiang, Yixing and Carducci, Valentina and Wyss, Richard and Desai, Rishi J and Alsentzer, Emily and Celi, Leo Anthony and Rodman, Adam and Schneeweiss, Sebastian and Chen, Jonathan H. and Romero-Brufau, Santiago and Lin, Kueiyu Joshua and Yang, Jie},
    year={2025},
    journal={arXiv preprint arXiv: 2504.19467},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2504.19467},
}
@article{clinical-text-review,
    title={Clinical text datasets for medical artificial intelligence and large language models—a systematic review},
    author={Wu, Jiageng and Liu, Xiaocong and Li, Minghui and Li, Wanxin and Su, Zichang and Lin, Shixu and Garay, Lucas and Zhang, Zhiyun and Zhang, Yujie and Zeng, Qingcheng and Shen, Jie and Yuan, Changzheng and Yang, Jie},
    journal={NEJM AI},
    volume={1},
    number={6},
    pages={AIra2400012},
    year={2024},
    publisher={Massachusetts Medical Society}
}

If you use the datasets in BRIDGE, please also cite the original paper of datasets, which can be found in our BRIDGE paper.