SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions

Abstract

Editing the figures in a research paper is a routine and time-consuming part of everyday research practice: authors relabel components, rearrange panels, and restyle visuals as they revise their manuscripts. Automating this editing workflow under a natural-language instruction, however, is challenging, because a scientific figure is a dense infographic in which heterogeneous visual elements such as schematics, plots, photos, captions, and arrows are composed under a tight visual grammar to advance a specific argument. To address this, we present SciDiagramEdit, a benchmark and skill-evolution framework that learns from natural paper revisions and operates on the figure's editable vector source, where users can inspect and co-edit individual primitives alongside the agent. Our benchmark mines before/after figure pairs from arXiv version histories, each grounded in the authors' own revision intent. To accommodate the diversity of editing instructions, we adopt agentic learning via skill evolution: an agentic proposer continually refines the agent's skill specification from execution traces over multiple epochs. The resulting skill progressively lifts edit accuracy on a held-out validation set, providing evidence that natural paper revisions are an effective training signal for instruction-driven figure editing.

Technical Analysis & Implementation

SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions§

Introduction§

Editing scientific figures is a routine but tedious task. SciDiagramEdit proposes a benchmark and framework that automates this process using natural-language instructions. The key idea is to learn from before/after figure pairs mined from arXiv version histories, which come with authors' revision intents. The framework operates on editable vector source figures and uses an agentic learning approach to continuously refine editing skills.

Methodology§

Benchmark Construction

The benchmark mines figure pairs from arXiv paper revisions. Each pair consists of a before-figure and an after-figure, along with the author's revision intent expressed in natural language. Figures are stored as editable vector graphics (e.g., SVG) where individual primitives (paths, text, arrows) can be manipulated.

Agentic Learning via Skill Evolution

The core innovation is an agentic proposer that maintains a skill specification—a set of high-level editing actions (e.g., "relabel axis", "change color", "reorder panels"). At each iteration, an agent executes the skill on a figure, generating an execution trace. The proposer then analyzes the trace (including success/failure feedback) to refine the skill specification. This process repeats over multiple epochs, progressively improving edit accuracy.

Mathematically, the skill specification $\mathcal{S}_t$ at time $t$ is updated by:

$$\mathcal{S}_{t+1} = \text{Proposer}\left(\mathcal{S}_t, \mathcal{T}_t, \mathcal{R}_t\right)$$

where $\mathcal{T}_t$ is the execution trace and $\mathcal{R}_t$ is the reward (e.g., edit accuracy on validation set).

Implementation Details§

The editing agent is a neural network that takes a figure (rendered as a set of primitive features) and an instruction, and outputs a sequence of edits (e.g., move, rotate, delete, add primitive). The agent is trained via reinforcement learning with the reward signal from the environment (whether the edited figure matches the target).

Code Snippet (PyTorch-like)

import torch
import torch.nn as nn

class EditingAgent(nn.Module):
    def __init__(self, primitive_dim=128, hidden_dim=256):
        super().__init__()
        self.fig_encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=primitive_dim, nhead=4), num_layers=2
        )
        self.text_encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=4), num_layers=2
        )
        self.action_decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model=hidden_dim, nhead=4), num_layers=2
        )
        self.action_head = nn.Linear(hidden_dim, num_actions)

    def forward(self, fig_primitives, instruction_tokens):
        fig_feats = self.fig_encoder(fig_primitives)
        text_feats = self.text_encoder(instruction_tokens)
        fused = self.action_decoder(text_feats, fig_feats)
        return self.action_head(fused)

Results§

Experiments show that the skill evolves over epochs, with increasing edit accuracy on a held-out validation set. The framework leverages natural paper revisions as an effective training signal, reducing the need for manual annotation.

Conclusion§

SciDiagramEdit demonstrates that figure editing can be automated by learning from paper revision histories, using an agentic skill-evolution framework that operates on vector graphics.