ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Technical Breakdown§

Core Methodology§

ClinFusion proposes a compositional cascaded vision encoder that processes heterogeneous 2D and 3D medical images through a shared pipeline. The key innovation is the Cascade Spatial-Aware Locality Fusion (CSLF) operator, which sequentially aggregates spatial features across scales and modalities.

Let $X \in \mathbb{R}^{C \times D \times H \times W}$ denote a 3D volume (or 2D slice with $D=1$). The encoder first extracts local features via a 3D convolutional stem:

$$F_0 = \text{Conv3D}(X) \in \mathbb{R}^{C' \times D' \times H' \times W'}$$

Then, $N$ cascaded CSLF blocks compute:

$$F_{i+1} = \text{CSLF}(F_i) = \text{MLP}(\text{Attention}(\text{LocalityPool}(F_i))) + F_i$$

where LocalityPool performs depthwise-separable 3D convolutions with kernel size $k$ and stride $s$, reducing spatial dimensions while preserving local context. Attention is a multi-head self-attention with relative positional bias. The cascaded design allows capturing both fine-grained anatomy and global context.

For 2D inputs, the 3D kernels are adapted via depthwise factorization, allowing weight sharing.

Vision-Grounded Evaluation§

• MedIF-Bench: A benchmark of 1,000 instruction-following pairs covering 10 radiology tasks (e.g., abnormality detection, differential diagnosis). Evaluates adherence to radiologist-style instructions.
• RoI-grounded metric: Given a report and ROI masks, computes alignment of generated text with radiologist-annotated regions via a cross-attention score:

$$\text{RoI-Score} = \frac{1}{|R|} \sum_{r \in R} \max_{t \in T_r} \text{sim}(e_t, e_{\text{ROI}_r})$$

where $e_t$ and $e_{\text{ROI}_r}$ are embeddings from a medical vision-language model, and $T_r$ are tokens describing region $r$.

Code Snippet (Illustrative PyTorch for CSLF Block)§

import torch.nn as nn

class LocalityPool3D(nn.Module):
    def __init__(self, dim, kernel_size=3, stride=2):
        super().__init__()
        self.conv = nn.Conv3d(dim, dim, kernel_size, stride=stride, groups=dim, padding=kernel_size//2)
        self.norm = nn.BatchNorm3d(dim)
    def forward(self, x):
        return self.norm(self.conv(x))

class CSLFBlock(nn.Module):
    def __init__(self, dim, num_heads=8):
        super().__init__()
        self.pool = LocalityPool3D(dim)
        self.attn = nn.MultiheadAttention(dim, num_heads, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(dim, 4*dim), nn.GELU(), nn.Linear(4*dim, dim))
        self.norm1 = nn.LayerNorm(dim)
        self.norm2 = nn.LayerNorm(dim)

    def forward(self, x):
        B, C, D, H, W = x.shape
        x_pool = self.pool(x)  # downsampled
        B2, C2, D2, H2, W2 = x_pool.shape
        x_flat = x_pool.flatten(2).transpose(1, 2)  # (B, L, C)
        attn_out, _ = self.attn(x_flat, x_flat, x_flat)
        attn_out = attn_out.transpose(1, 2).view(B, C2, D2, H2, W2)
        attn_out = self.norm1(attn_out.permute(0,2,3,4,1)).permute(0,4,1,2,3)
        out = self.mlp(attn_out.permute(0,2,3,4,1)).permute(0,4,1,2,3)
        out = self.norm2(out.permute(0,2,3,4,1)).permute(0,4,1,2,3)
        return out + attn_out  # residual

Agentic Augmentation§

ClinFusion can integrate retrieval-augmented generation (RAG) and tool use (e.g., calling a segmentation model for ROI extraction), forming an agentic workflow for clinical decision support.

Results§

• State-of-the-art on 20/24 medical benchmarks (including VQA, report generation, instruction following).
• Outperforms GPT-5.2 and Gemini-3-Flash on 13/16 benchmarks.
• Radiologist evaluation confirms highest report quality and strong correlation of RoI-grounded metric with expert judgment.