Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

Technical Summary§

This paper targets cross-skill long-horizon reasoning—multi-step tasks where each step requires a different skill and depends on earlier outputs. The authors propose a principled measure, a benchmark, and a training method to improve skill switching in LLMs.

Skill Entropy§

For a task with a gold skill sequence $s_1, s_2, \dots, s_T$, the task-level skill entropy is defined over transition probabilities estimated from a large corpus of reasoning chains:

$$H(s_{1:T}) = -\frac{1}{T-1} \sum_{t=1}^{T-1} \log P(s_{t+1} \mid s_t).$$

Higher entropy indicates harder skill transitions. The Skill^2-Bench benchmark contains tasks over 558 skills across 9 verifiable domains, each annotated with gold skill sequences and grouped into low, medium, and high entropy difficulty levels. Evaluation of 8 frontier and 4 open-source models reveals a skill-switching gap: accuracy drops monotonically as task entropy increases.

Skill-Entropy RL§

The training framework augments the base LLM with an additional skill-prediction head. At each reasoning step $t$, the policy outputs both the next answer token $a_t$ and a skill label $\hat{s}_t$. The reward combines step-level correctness with a skill-entropy alignment reward:

$$R = \sum_{t=1}^T \left[ r_{\text{correct}}(a_t) + \lambda \cdot r_{\text{skill}}(\hat{s}_t, s_t) \right],$$

where $r_{\text{skill}}$ is the negative cross-entropy between the predicted skill distribution and the gold skill label, explicitly rewarding the model for planning its cognitive moves. The gradient is computed with a policy-gradient objective:

$$\nabla_\theta J = \mathbb{E}_\pi \left[ \nabla_\theta \log \pi_\theta(a_t, \hat{s}_t \mid x_{<t}) \cdot R \right].$$

Implementation Sketch§

A simplified PyTorch training loop:

import torch.nn.functional as F

for input_ids, gold_answer, gold_skill in data_loader:
    # policy with two heads: answer and skill
    answer_logits, skill_logits = policy(input_ids)
    
    # sample action and predict skill
    answer_sample = answer_logits.argmax(-1)
    skill_pred = skill_logits.argmax(-1)
    
    # step-level rewards
    r_correct = (answer_sample == gold_answer).float()
    r_skill = -F.cross_entropy(skill_logits, gold_skill, reduction='none')
    reward = r_correct + lambda_skill * r_skill
    
    # policy gradient loss
    log_prob = policy.log_prob(answer_sample, skill_pred, input_ids)
    loss = -(log_prob * reward).mean()
    loss.backward()

Results§

On Qwen3-4B-Instruct, Skill-Entropy RL improves Skill^2-Bench accuracy from 34.4% to 68.4%; on Qwen3-1.7B, from 14.6% to 40.1%, outperforming competitive RL baselines. The same pipeline transfers to off-the-shelf datasets like OpenR1-Math, demonstrating that skill entropy is a reusable training signal. Code is available at the project repository.