PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

Overview§

SWE-bench-like benchmarks pair a PR with its linked issue to evaluate LLM issue resolution. However, such pairings are often misaligned in practice. The authors analyze SWE-bench Verified and find 13.6% of instances exhibit misalignment across five patterns and eleven fine-grained scenarios. To address this, they propose PAIChecker, a multi-agent system that automatically checks PR-Issue alignment in a scalable and reliable manner.

Methodology§

PAIChecker adopts a three-phase design:

1. Specific Pattern Identification: Specialized agents independently detect known misalignment patterns (e.g., missing issue references, unrelated patch content). Each agent focuses on a distinct pattern and outputs a candidate label.
2. Cross-Agent Label Synthesis: A synthesis agent aggregates the individual agents' labels, weighing their confidence to produce a combined raw prediction. This helps generalize to unseen scenarios.
3. Code-Level Validation: The raw prediction is validated by executing the PR patch against the issue's tests. A validation agent determines if the patch actually resolves the issue, thereby reducing false positives.

The misalignment probability is modeled as a weighted combination: $$P(\text{misaligned}) = \sigma\left(\sum_{i=1}^{N} w_i \cdot \ell_i + \beta \cdot v\right)$$ where $\ell_i$ is the label from pattern agent $i$, $w_i$ is its learned weight, $v \in \{0,1\}$ is the code-validation outcome, and $\sigma$ is a logistic function.

Implementation§

The system is built on a multi-agent LLM framework. Each agent runs a different prompt template tailored to a misalignment pattern. Below is an illustrative Python sketch:

from typing import List

def pai_checker(pr_data, issue_data, llm):
    patterns = ["missing_ref", "unrelated_patch", "wrong_fix", "partial_fix", "ambiguous_issue"]
    agent_votes = []
    for p in patterns:
        prompt = build_pattern_prompt(p, pr_data, issue_data)
        vote = llm.generate(prompt)  # returns {0,1}
        agent_votes.append(vote)
    
    synthesis_prompt = build_synthesis_prompt(agent_votes)
    raw_pred = llm.generate(synthesis_prompt)
    
    test_result = run_tests(pr_data["patch"], issue_data["tests"])
    validation_prompt = build_validation_prompt(raw_pred, test_result)
    final_pred = llm.generate(validation_prompt)
    return final_pred

Key implementation details include:

• LLM backbones: GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, DeepSeek-V2.5 are used as the underlying agents.
• Pattern taxonomy: Five patterns are decomposed into eleven fine-grained scenarios to improve detection granularity.
• Test oracle integration: The code-level validation phase reuses the SWE-bench test harness to check if the patch passes the issue's associated tests.

Results§

The authors evaluate PAIChecker on SWE-Gym and SWE-bench Multilingual. Across all four LLM backbones, PAIChecker achieves the best binary accuracy compared to baseline methods, reaching up to 92.12% on SWE-Gym and 91.67% on SWE-bench Multilingual. Ablation studies confirm that each phase contributes positively, with the code-level validation providing the largest gain by eliminating false positives.

Conclusion§

PAIChecker is a practical multi-agent system for ensuring the reliability of SWE-bench-like benchmarks. Its three-phase design enables accurate, generalizable, and progressively verified misalignment detection, making it a valuable tool for the LLM evaluation community.