MedVTok Code

A general-purpose medical visual tokenizer

MedVTok

Medical imaging, unified at the token level.

A shared foundation for medical images and volumes, preserving anatomical structure and clinical meaning across modalities.

Explore MedVTok Model weights Code Paper Soon
2D images & 3D volumesA spectrum of medical imaging

01 / The interface

Many modalities, one interface.

Medical imaging offers complementary views of the body. MedVTok encodes 2D images and 3D volumes across nine modalities into a common token interface that downstream models can reuse.

Diverse visual evidence

CT
Histopathology
MRI
Fundus
Endoscopy
X-ray
Dermoscopy
Ultrasound
OCT

9 modalities · 2D images & 3D volumes

Image patches

Unified medical visual tokenizer

MedVTok

Visual tokens

Anatomical structure · Clinical semantics

Downstream tasks

Frozen tokenizer · task-specific models

02 / The approach

Many experts, one tokenizer.

Different medical imaging experts and foundation models capture complementary knowledge. MERA (Multi-Expert Representation Alignment) lets MedVTok learn from the specialist matched to each input. Shared-space alignment brings token and expert features together; alignment in the expert’s original space helps retain its distinct knowledge.

Alignment Alignment
CT
MedVTok
Token features
Shared space
Multi-expert panel
Pathology expertFrozen parameters
MRI expertFrozen parameters
CT expertFrozen parameters
Fundus expertFrozen parameters
Projection

03 / The corpus

Training corpus.

We train this shared foundation on over 28 million 2D images and 70,000 3D volumes spanning nine imaging modalities.

Modality 28M+2D images 70K+3D volumes
CT
9.26M32.2%
35.3K50.3%
Histopathology
5.65M19.6%
 —
MRI
3.79M13.2%
34.9K49.7%
Fundus
3.05M10.6%
 —
Endoscopy
2.86M9.9%
 —
X-ray
1.59M5.5%
 —
Dermoscopy
1.26M4.4%
 —
Ultrasound
1.00M3.5%
 —
OCT
317.8K1.1%
 —

Rounded sample counts · Percentages within each dimension · — Not included

04 / Token fidelity

Preserve the visual evidence.

Reconstructing the input tests what the tokens preserve. MedVTok achieves the highest PSNR and SSIM among the compared tokenizers across nine 2D modalities and native 3D CT and MRI.

dB ↑
Reconstruction benchmark scores by model
Benchmarks MedVTok PUMIT MedVAE MedITok AToken
2D FundusJSIEC 42.22 32.03 17.11 35.92 41.08
2D CTChestCTScan 44.75 32.86 37.97 35.61 41.60
2D DermoscopyHAM10K 40.39 27.05 22.42 33.81 39.00
2D EndoscopyPolypDiag 41.42 28.73 16.83 32.42 39.69
2D MRIBRISC 43.75 32.88 35.72 34.82 41.64
2D OCTOCTDL 35.04 27.06 30.90 30.26 32.94
2D HistopathologyLungHist700 30.16 19.43 20.13 26.03 28.57
2D UltrasoundBUSBRA 42.08 23.26 33.47 32.62 39.41
2D X-rayKermany 41.83 26.69 37.04 34.70 40.02
3D MRIMRISegmenter 31.77 29.96 25.89 — 31.46
3D CTAMOS-MM 36.50 31.87 31.30 — 35.68

↑ Higher is better. Bold marks the best score in each row. — Not reported.

Benchmark data

05 / Downstream utility

One frozen tokenizer, many tasks.

We then test how well the representation transfers. With MedVTok frozen, task-specific models use its features for classification, retrieval, segmentation, image synthesis, visual question answering, and report generation.

Downstream benchmarks
Downstream benchmark scores by model
Benchmarks MedVTok PUMIT MedVAE MedITok
ClassificationJSIEC · ChestCTScan · HAM10K · PolypDiag · BRISC · OCTDL · LungHist700 · BUSBRA · KermanyMean · Accuracy (%) ↑ 80.23 52.41 51.32 73.55
RetrievalJSIEC · ChestCTScan · HAM10K · PolypDiag · BRISC · OCTDL · LungHist700 · BUSBRA · KermanyMean · Recall@1 (%) ↑ 79.18 52.53 51.30 74.43
SegmentationMaplesDR · MosMedDataPlus · HAM10K · PolypDB · BRISC · PanNuke · BUSBRA · VinDr-RibCXRMean · Dice (%) ↑ 76.07 53.63 38.86 71.99
Image synthesisGMAI-VL-5.5MMean · FID ↓ 101.73 227.51 172.08 109.76
Visual question answeringFundus-MMBench · MM-Skin · Kvasir-VQA-x1 · PathVQA · SLAKE · GMAI-MMBenchMean · Accuracy (%) ↑ 46.48 41.94 39.93 45.39
Report generationAMOS-MMMean · GREEN ↑ 0.413 0.250 0.230 0.386

Paper-reported scores with the tokenizer frozen. ↑ Higher is better; ↓ lower is better. Bold marks the best score in each row.

Benchmark data

06 / Looking ahead

Toward a health world model.

MedVTok is a starting point for a broader goal: a coherent representation of a person’s health across modalities and over time.

We aim to connect medical images with clinical records and everyday health signals in a health world model that evolves as new observations arrive, supporting both patient care and lifelong well-being.

Model weights Source code 2D & 3D Demo

The team

MedVTok

A General-Purpose Medical Visual Tokenizer

Chenglong Ma1,2,* Yuanfeng Ji3,7,* Junzhi Ning4,* Jiyao Liu1,* Yujie Wei1 Shaohao Rui2,6 Chaoyang Zhang2 Wenjie Li2,6 Wenhao Tang4 Cheng Tang4 Wei Li4,6 Shujian Gao1,4 Yujie Zhang2,6 Ying Chen4 Zilong Li1 Zongyuan Ge5 Diping Song4 Jin Ye4,5 Ming Hu4,5 Junjun He2,4,7,† Hongming Shan1,†

* Equal contribution † Corresponding authors

Author affiliations
1 Fudan University 2 Shanghai Innovation Institute 3 Stanford University School of Medicine 4 Shanghai Artificial Intelligence Laboratory 5 Monash University 6 Shanghai Jiao Tong University 7 Ant Healthcare, Ant Group

Acknowledgments

We thank the teams behind DermLIP, RET-CLIP, CONCH v1.5, PeskaVLP, MedImageInsight, SPECTRE, HLIP, MedVAE, and AToken for their open-source contributions. The code for training MedVTok is adapted from MedITok and UniTok.

Illustrative image credits

The hero combines illustrative modality images and matched evaluation examples. CT and MRI orthogonal views come from 3D Slicer sample data. Reconstruction galleries use matched evaluation outputs. Synthesis, VQA and report examples are adapted from the paper.

Open full-resolution figure ↗