From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation

Overview§

This paper reframes dataset construction for generalist image generation as a capability-co-evolution problem. Rather than optimizing each task corpus independently, the authors build a unified data infrastructure that couples capability-specific supervision with a curriculum that follows the natural dependency order of generative capabilities. The result is a 440M-image T2I corpus, 120M editing pairs, and 27M image-entity pairs used to train 3B and 6B diffusion models from scratch.

Core Methodology§

Three specialized data engines construct complementary relational supervision:

1. Text-image grounding engine: curates high-quality image-text pairs with diverse visual concepts and caption granularities.
2. Inter-image transformation engine: generates editing pairs (source, instruction, target) that teach localized/global image transformations.
3. Image-knowledge association engine: ties images to entities and descriptive knowledge, enabling world-knowledge grounded generation.

Caption experts align T2I and editing supervision across tasks and granularities, ensuring consistent conditioning. The multi-stage curriculum jointly evolves task composition, visual-concept distribution, data quality, and image resolution along the dependency order of capability acquisition. If $\mathcal{D}_{cap}$ is the dataset for capability $cap$, the curriculum defines a task-sampling distribution $\pi_t$ at step $t$ based on measured capability gaps: $\pi_t \propto \exp(gap_t / \tau)$. A capability-aware evaluation loop closes the gap via targeted retrieval, expert construction, and gap-aware resampling.

The diffusion model is trained with the standard denoising objective:

$$L = \mathbb{E}_{t, \epsilon} \left\| \epsilon - \epsilon_\theta(x_t, t, c) \right\|^2$$

For editing, the conditioning $c$ concatenates the text embedding of the edit instruction with the source image encoding: $c = [\text{CLIP}(e); \text{CLIP}(x_0)]$. For knowledge grounding, $c$ includes entity descriptions retrieved from the image-knowledge engine.

Implementation Details§

The framework uses a multi-stage schedule where capabilities are acquired in order: grounding first, then transformation, then knowledge association. Data quality and resolution are increased as training progresses. For example, T2I data starts at 256px and later incorporates 512px and 1024px samples; editing pairs start with simple attribute changes and escalate to complex structural edits.

A simplified PyTorch training loop is shown below:

# Training loop with capability scheduling
for step, batch in enumerate(train_loader):
    task = curriculum.sample_task(step)
    if task == 't2i':
        x, cond = batch['image'], batch['caption']
        loss = diffusion_loss(model, x, cond)
    elif task == 'edit':
        x, cond = batch['target'], batch['instr'] + batch['source']
        loss = diffusion_loss(model, x, cond)
    elif task == 'knowledge':
        x, cond = batch['image'], batch['entity_description']
        loss = diffusion_loss(model, x, cond)
    loss.backward()
    optimizer.step()
    if step % eval_interval == 0:
        gaps = evaluate_capabilities(model, eval_bench)
        curriculum.update(task_probs, gaps)

The backbone is a diffusion transformer with 3B or 6B parameters. The data infrastructure is modular; each engine can be upgraded or resampled based on evaluation feedback. This closed-loop design is key to co-evolving capabilities rather than merely concatenating static corpora.

Results§

Quantitative evaluation on CPI-Bench, combined with qualitative tests, shows broad visual coverage, versatile rendering, and effective transfer across generative capabilities. The authors demonstrate that their capability-centric data organization outperforms task-isolated curation baselines, especially in editing and knowledge-grounded generation scenarios.