G-CARL: Grounded Checklist-Aligned Reward Learning for Patient-Oriented Medical Report Interpretation

Technical Breakdown: G-CARL§

G-CARL addresses the Patient-oriented Medical Report Interpretation (PMRI) task, where a model must generate an accessible explanation of a medical report conditioned on a user's query and dialogue history. The core challenge is jointly optimizing for evidence-grounded medical factuality (verifiable against the report) and context-dependent patient communication (subjective and variable).

Method Overview§

The authors propose a grounded, checklist-aligned reinforcement learning framework. Instead of relying on holistic reward models or supervised fine-tuning alone, G-CARL decomposes supervision into three structured components:

1. Atomic Claim Verification (Factuality): The generation is split into atomic claims. Each claim is verified against the medical report and retrieved external knowledge via a multi-source retrieval system. A claim-level factual reward $r_{fact}$ is computed based on whether the claim is entailed by the retrieved evidence.
2. Context-Aware Checklist Coverage (User-Demand): An instance-specific weighted checklist is dynamically generated based on the user query and dialogue history. The coverage reward $r_{cov}$ measures how many required items (e.g., diagnosis explanation, treatment advice, side effects) are present in the response, with weights reflecting patient context.
3. Expression Quality: A language quality reward $r_{qual}$ (e.g., via GPT-4 scoring or n-gram metrics) encourages coherence and readability.

The total reward is a weighted sum:

$$R = \lambda_f \, r_{fact} + \lambda_c \, r_{cov} + \lambda_q \, r_{qual}$$

Training with Reinforcement Learning§

G-CARL uses a policy-gradient approach (e.g., PPO) to fine-tune a pretrained multimodal model (e.g., a vision-language model backboned by a LLM). The model receives the medical image (e.g., X-ray) and the patient query as input. The training loop:

• Generate a response from the current policy.
• Extract atomic claims (via an LLM-based decomposer).
• Retrieve evidence from a knowledge base (e.g., medical literature, report text) to compute claim accuracy.
• Compute checklist coverage by matching the response to a clinician-designed checklist.
• Combine rewards and update the policy via PPO with a KL penalty to the reference model.

Implementation Sketch§

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoProcessor

class GCARLTrainer:
    def __init__(self, base_model, ref_model, retriever, checklist_generator, eps=0.2):
        self.policy = base_model
        self.ref_model = ref_model
        self.retriever = retriever
        self.checklist_gen = checklist_generator
        self.eps = eps

    def compute_reward(self, response, report_text, query, context):
        # Decompose into claims
        claims = self.extract_claims(response)  # e.g., using an LLM
        # Multi-source retrieval for each claim
        evidence = [self.retriever.retrieve(claim) for claim in claims]
        # Factual precision: proportion of claims entailed by evidence
        fact_reward = sum(self.check_entailment(c, e) for c, e in zip(claims, evidence)) / len(claims)
        # Dynamic checklist coverage
        checklist = self.checklist_gen(query, context)
        coverage = sum(1 for item in checklist if item in response) / len(checklist)
        # Expression quality via e.g., ROUGE or neural scoring
        quality = self.quality_score(response, query)
        return 0.5 * fact_reward + 0.3 * coverage + 0.2 * quality

    def ppo_update(self, batch):
        # Standard PPO objective with clipping and KL penalty
        for (query, image, report, response, old_log_prob, reward) in batch:
            new_log_prob = self.policy(query, image).log_prob(response)
            ratio = torch.exp(new_log_prob - old_log_prob)
            clip_ratio = torch.clamp(ratio, 1 - self.eps, 1 + self.eps)
            kl = F.kl_div(new_log_prob, self.ref_model(query, image).log_prob(response), reduce='sum')
            loss = -torch.min(ratio * reward, clip_ratio * reward) + 0.01 * kl
            loss.backward()

Key Insights§

• The use of instance-specific weighted checklists aligns generation with user needs without constraining diversity (unlike fixed templates).
• Multi-source retrieval provides grounded evidence for atomic claim verification, which is more reliable than a single holistic reward model.
• The approach decouples verifiable factuality from subjective communication quality, enabling joint optimization without catastrophic forgetting.

Results§

The authors introduce MMedReport, a real-world benchmark, and evaluate with a clinician-designed 3D protocol. G-CARL consistently outperforms supervised fine-tuning and other RL baselines in overall quality, claim-level precision, and checklist recall, as confirmed by clinician pairwise preference.