Multi-Agent AI System for Radiology Report Structuring and Quality Assurance with Independent Radiologist Evaluation

Overview§

The paper presents a multi-agent AI system for radiology report structuring and quality assurance (QA), deployed locally to preserve patient privacy. The pipeline processes free-text CT reports (chest, abdomen, pelvis) and transforms them into a standardized anatomical section layout while simultaneously detecting common errors. A retrospective set of 638 reports from 15 radiologists was used; two independent radiologists evaluated a 45-report subset.

Methodology§

Report Structuring§

The system operates at the sentence level. First, regex-based rules split reports into sentences and classify them into anatomical categories (e.g., liver, pancreas, lungs). Local large language models (LLMs) then refine this classification, handling ambiguous phrasing and ensuring consistency. Each sentence is assigned a section header from a predefined ontology. This is a two-stage process:

1. Rule-based pre‑segmentation: Regex patterns identify sentence boundaries and initial category candidates.
2. LLM-based refinement: A local LLM (e.g., a quantized 7B model) re-ranks or corrects the regex assignments using a prompt engineered with the report context.

Formally, for a sentence $s_i$, the system chooses the section $\hat{y}_i$ that maximizes the probability under the combined model:

$$ \hat{y}_i = \arg\max_{y \in \mathcal{Y}} \; \big( \lambda \cdot P_{\text{regex}}(y|s_i) + (1-\lambda) \cdot P_{\text{LLM}}(y|s_i, \text{context}) \big) $$

where $\lambda$ is a tunable weight, $\mathcal{Y}$ is the set of anatomical sections, and the context includes neighboring sentences and the report's overall impression section.

Quality Assurance (QA)§

The QA agent is composed of three specialized sub-agents:

• Mismatch detector: Compares findings and impression for contradictions or omissions. It uses an LLM to generate semantic embeddings of sections and computes cosine similarity to flag inconsistencies.
• Gender-anatomy conflict detector: Checks for references to organs inconsistent with the reported patient sex (e.g., prostate in a female patient) via rule-based keyword matching plus LLM verification.
• Critical findings communicator: Detects phrases indicating emergent findings (e.g., "pneumothorax") and checks whether the report mentions communication to the referring physician. If missing, the system flags it.

Each sub-agent outputs a binary flag and a confidence score. An overall risk score is computed as a weighted sum:

$$ \text{QA}_\text{score} = w_1 \cdot \text{mismatch} + w_2 \cdot \text{gender} + w_3 \cdot \text{communication} $$

Evaluation§

Two board-certified radiologists independently reviewed 45 randomly selected reports. They rated restructuring correctness and QA performance. The primary metrics were inter-reviewer agreement and the proportion of reports deemed correctly structured.

Results§

• Structuring: All 22,270 sentences were successfully structured. The two reviewers agreed that 31/45 (69%) reports were correctly restructured, while 2 (4%) were incorrect; they disagreed on 12 (27%). No clinically important content was lost or fabricated.
• QA flags: 90/638 (14.1%) reports were flagged; the most common issue was section mismatches (80 reports, 12.5%).
• QA quality: Reviewers rated 84% of flagged reports as "excellent" or "good" and the rest as "fair".

Implementation Sketch§

The following Python snippet outlines the core multi-agent pipeline:

import re
import json
from typing import List, Dict

# Assume a local LLM interface (e.g., Ollama, llama.cpp)
class LocalLLM:
    def complete(self, prompt: str) -> str:
        # In practice, calls a local model
        return "liver"  # dummy response

class ReportStructuringAgent:
    def __init__(self, llm: LocalLLM):
        self.llm = llm
        self.section_map = {"liver": "Hepatobiliary", "pancreas": "Pancreas"}

    def split_sentences(self, text: str) -> List[str]:
        return re.split(r'(?<=[.!?])\s+', text)

    def extract_sections(self, report: str) -> Dict[str, List[str]]:
        structured = {}
        for sentence in self.split_sentences(report):
            # Step 1: regex heuristic
            regex_label = self.regex_label(sentence)
            # Step 2: LLM refinement
            llm_label = self.llm.complete(f"Classify this sentence: {sentence}")
            label = self.resolve_label(regex_label, llm_label)
            structured.setdefault(label, []).append(sentence)
        return structured

    def regex_label(self, sent: str) -> str:
        if re.search(r'liver|hepatic', sent, re.I):
            return "liver"
        return "other"

    def resolve_label(self, regex_label: str, llm_label: str) -> str:
        return regex_label if regex_label != "other" else llm_label

class QAAgent:
    def __init__(self, llm: LocalLLM):
        self.llm = llm

    def check_mismatch(self, findings: str, impression: str) -> bool:
        # Use embeddings from LLM to compare semantics
        f_vec = self.llm.embed(findings)
        i_vec = self.llm.embed(impression)
        cos_sim = dot(f_vec, i_vec) / (norm(f_vec) * norm(i_vec))
        return cos_sim < 0.8

# Pipeline orchestration
llm = LocalLLM()
struct_agent = ReportStructuringAgent(llm)
qa_agent = QAAgent(llm)

def run_pipeline(report_text: str) -> Dict:
    structured = struct_agent.extract_sections(report_text)
    findings = " ".join(structured.get("Findings", []))
    impression = " ".join(structured.get("Impression", []))
    mismatch = qa_agent.check_mismatch(findings, impression)
    return {
        "structured": structured,
        "qa_flags": {"mismatch": mismatch}
    }

Key Takeaways§

• Combining rule-based and LLM-based approaches improves reliability while keeping compute local and private.
• The multi-agent design separates concerns (structuring vs. QA), making each component independently testable and replaceable.
• The modest 14.1% flag rate and high radiologist agreement indicate practical utility for real‑time clinical workflow support.