Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills

Overview§

Skill Self-Play (Skill-SP) is a co-evolutionary framework for LLM training that bridges the gap between structured verification and open-ended exploration. It consists of three components: a proposer, a solver, and a skill controller. These components interact in a continuous self-play loop, where the proposer generates challenging tasks conditioned on dynamically sampled skills, the solver explores solutions, and the skill controller updates and expands the skill library based on execution feedback.

Methodology§

The core idea is to represent each skill as a specialized policy or prompt that ensures deep, verifiable execution in a specific scenario. The skill library $\mathcal{S} = \{s_1, s_2, \dots, s_K\}$ is maintained and updated. At each iteration $t$:

1. Proposer: Given the current skill library, the proposer $\pi_{\text{prop}}$ samples a set of skills $\mathcal{S}_t$ and generates a task $x_t$ that requires the coordinated use of these skills: $$x_t \sim \pi_{\text{prop}}(\cdot | \mathcal{S}_t)$$

2. Solver: The solver $\pi_{\text{sol}}$ receives the task and attempts to produce a solution $y_t$: $$y_t \sim \pi_{\text{sol}}(\cdot | x_t)$$

3. Skill Controller: The task and solution are evaluated against a verifier $V$ that gives a binary reward $r_t = V(x_t, y_t)$. The controller then updates the skill library using a reinforcement learning rule: $$\mathcal{S}_{t+1} = \mathcal{S}_t \cup \{ \text{update}(s, r_t) \}$$ where update may add new skills derived from successful trajectories or refine existing ones.

Training Loop§

The entire system is optimized via a reinforcement learning loop that maximizes the expected reward: $$\mathcal{J} = \mathbb{E}_{x \sim \pi_{\text{prop}}, y \sim \pi_{\text{sol}}}[V(x,y)]$$

Both the proposer and solver are trained using policy gradient methods (e.g., PPO) to improve their policies based on the rewards. The skill controller uses success/failure feedback to adjust the skill library; skills that are frequently used in successful tasks are reinforced, while redundant or ineffective skills are pruned.

Implementation Details§

• The proposer is an LLM that takes a prompt describing available skills and outputs a task description.
• The solver is an LLM (potentially the same base model) that generates step-by-step reasoning or tool calls.
• The verifier can be a rule-based checker (e.g., for tool-use) or a learned reward model.
• The skill library is stored as a set of few-shot examples or natural language descriptions.

Code Snippet§

Below is a simplified PyTorch-like pseudocode illustrating the core training loop:

class SkillSPTrainer:
    def __init__(self, proposer, solver, skill_library, verifier, lr=1e-5):
        self.proposer = proposer
        self.solver = solver
        self.skill_library = skill_library
        self.verifier = verifier
        self.optimizer = torch.optim.Adam(
            list(self.proposer.parameters()) + list(self.solver.parameters()), lr=lr)

    def train_step(self):
        # 1. Sample skills and generate task
        skills = random.sample(self.skill_library, k=2)
        task = self.proposer.generate(skills)

        # 2. Solve task
        solution = self.solver.generate(task)

        # 3. Verify
        reward = self.verifier.check(task, solution)

        # 4. Update skill library (simplified)
        if reward > 0:
            self.skill_library.add(f"skill_{len(self.skill_library)}")

        # 5. RL update (PPO loss simplified)
        log_prob_sol = self.solver.log_prob(solution, task)
        log_prob_prop = self.proposer.log_prob(task, skills)
        loss = - (log_prob_sol + log_prob_prop) * reward
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

Key Insights§

• Skills act as a middle ground: each skill provides a verifiable execution scenario, while dynamic routing maintains task diversity.
• Co-evolution ensures that as the solver improves, the proposer generates harder tasks, preventing saturation.
• Empirical results on tool-use and reasoning benchmarks show consistent performance gains and even turnarounds for initially misaligned models.

Conclusion§

Skill Self-Play offers a principled way to balance exploration and verification in LLM self-evolution, outperforming prior open-ended and environment-bound methods.