Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Training

Overview§

Reasoning Core is a new collection of 50 procedural generators that produce verifiable reasoning problems across mathematics, logic, planning, state tracking, formal languages, structured data, games, causality, and code. The paper focuses on using these generators for completion-supervised fine-tuning (C-SFT) of language models, where the model learns to directly produce the final answer via standard language modeling loss. The key contribution is not just the dataset scale, but an analysis of what makes procedural data useful: compact targets and calibrated difficulty matter more than raw semantic validity.

Methodology§

Procedural Generators§

Each generator synthesizes a problem $x$ (e.g., a math word problem) and a target answer $y$ (a short string like 42). Generators expose three components:

• Semantic scorer: validates that $x$ and $y$ are consistent (e.g., arithmetic correctness).
• Difficulty controls: tunable parameters such as number of operations, grid size, or steps in a plan.
• Task evaluator: checks model predictions against canonical answers, often using exact or fuzzy matching.

Completion-Supervised Training§

Given a prompt template $T$, each example is formatted as $[T(x) \, \| \, y]$ and trained with the standard next-token prediction loss on the target tokens:

$$ \mathcal{L} = -\sum_{t=1}^{|y|} \log p_\theta(y_t \mid T(x), y_{<t}) $$

The comparison protocol matches training duration and model size across all datasets. In the primary 3B parameter setup, Reasoning Core achieves the highest mean scores on DROP, LogiQA, and ARC-Challenge, beating both a no-procedural baseline and existing procedural collections (Procedural Warmup, Reasoning Gym, SynLogic).

Why Some Generators Fail§

The paper finds that generators with high semantic validity can still degrade performance. Two factors stand out:

1. Target compactness: Generators producing verbose or multi-step explanations as targets hurt utility. Compact final answers force the model to internalize reasoning rather than parroting templates.
2. Difficulty calibration: If difficulty is too high or uncalibrated, models do not learn a smooth curriculum. Controlled difficulty (e.g., gradually increasing operation count) significantly improves generalization to unseen tasks.

Implementation Sketch§

A simplified view of data generation and training:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from reasoning_core import get_generator

# Example: arithmetic generator with difficulty control
gen = get_generator("arithmetic", max_ops=3, max_digits=4)

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)

for step in range(10000):
    x, y, meta = gen.generate(difficulty=0.7)  # calibrated difficulty
    prompt = f"Question: {x}\nAnswer:"
    inputs = tokenizer(prompt + y, return_tensors="pt")["input_ids"]

    # Completion-supervised loss: only compute on target tokens
    labels = inputs.clone()
    prompt_len = len(tokenizer(prompt)['input_ids'])
    labels[:, :prompt_len] = -100

    logits = model(inputs, labels=labels).logits
    loss = torch.nn.functional.cross_entropy(
        logits[:, prompt_len-1:-1].reshape(-1, logits.size(-1)),
        inputs[:, prompt_len:].reshape(-1)
    )
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Audits & Quality Control§

The authors perform rigorous audits combining model-assisted review (using a judge LLM to flag suspicious examples), human adjudication, and regression testing. These audits uncovered subtle bugs in generation, rendering, target construction, and scoring across all datasets, including their own. This underscores that procedural generation does not guarantee correctness—data pipelines need continuous validation.

Conclusion§

Reasoning Core demonstrates that large-scale procedural data can be highly effective for teaching reasoning via completion supervision, provided that targets are compact and difficulty is well-calibrated. The public release of generators, datasets, and audit materials should accelerate further research into automated reasoning data synthesis.