Skill-Space Shooting for Autonomous Robot Policy Improvement

Core Insight§

Robots must improve beyond initial training in the physical world. While foundation models can compose behaviors to complete tasks, they don't directly teach a policy to overcome its own failures. Skill-space shooting bridges this gap by treating corrective behaviors as reusable skills—short, recurring action sequences that foundation models can reason about from a scene. These skills become a source of corrective supervision, enabling scalable policy improvement within and across tasks.

Methodology§

Given a pretrained policy π and a set of skills S = {s₁, ..., sₖ} (e.g., "push left", "grasp handle"), the system alternates between:

1. Skill selection: A foundation model selects a skill s ∈ S and a shooting horizon H based on the current scene and task.
2. Trajectory generation: The robot executes the skill as a sequence of actions a_{t:t+H} = s(θ, x_t), where θ are skill parameters and x_t is the state.
3. Policy improvement: If the trial succeeds (task reward r > threshold), the trajectory is added to the policy's replay buffer. The policy is updated via behavior cloning or RL:

$$\mathcal{L}(\pi) = \mathbb{E}_{(x_t, a_t) \sim \mathcal{D}_{\text{success}}} \left[ -\log \pi(a_t | x_t) \right] + \lambda \mathcal{L}_{\text{reg}}$$ where $\mathcal{L}_{\text{reg}}$ regularizes against forgetting.

Key details:

• Skills are parameterized by a foundation model (e.g., LLM/VLM) that outputs low-level action primitives or waypoints.
• Failure cases trigger exploration: the foundation model proposes alternative skills from S, effectively performing shooting in skill space.
• Successful skill trajectories are stored and shared across tasks, reducing the need for new human demonstrations.
• The policy improvement is iterative: after each update, the policy is re-evaluated, and the cycle repeats.

Implementation§

A PyTorch-style pseudocode for the improvement loop:

import torch
import torch.nn as nn
import numpy as np

class SkillSpaceShooting:
    def __init__(self, policy, skills, foundation_model, lr=1e-4):
        self.policy = policy  # e.g., a neural network
        self.skills = skills  # list of skill functions
        self.fm = foundation_model  # e.g., LLM/VLM
        self.optimizer = torch.optim.Adam(policy.parameters(), lr=lr)
        self.buffer = []

    def select_skill(self, state, task_desc):
        # Foundation model picks a skill and horizon
        prompt = f"Task: {task_desc}. State: {state}. Available skills: {self.skills}. Choose skill and horizon."
        skill_id, horizon = self.fm.generate(prompt)
        return self.skills[skill_id], horizon

    def execute_skill(self, skill, horizon, state):
        # Rollout skill for horizon steps
        traj = []
        for _ in range(horizon):
            action = skill(state)  # skill maps state to action
            traj.append((state, action))
            state = self.env.step(action)
        success = self.env.check_success()
        return traj, success

    def update_policy(self, batch_size=32):
        if len(self.buffer) < batch_size:
            return
        batch = np.random.choice(self.buffer, batch_size)
        states = torch.stack([torch.tensor(s) for s, a in batch])
        actions = torch.stack([torch.tensor(a) for s, a in batch])
        pred_actions = self.policy(states)
        loss = nn.MSELoss()(pred_actions, actions)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

    def step(self, state, task_desc):
        skill, horizon = self.select_skill(state, task_desc)
        traj, success = self.execute_skill(skill, horizon, state)
        if success:
            self.buffer.extend(traj)
        self.update_policy()
        return success

Results and Implications§

Real-world experiments demonstrate repeated improvement of policies acting autonomously. Skills can be shared across tasks, reducing the teaching burden for new tasks. This approach enables scalable and generalizable policy improvement, leveraging foundation models as guides for corrective supervision without human demonstrations. The method is agnostic to the policy architecture and skill representation, making it broadly applicable to robotics and beyond.

Conclusion§

Skill-space shooting transforms reusable skills into a source of corrective supervision, enabling autonomous policy improvement. By combining foundation model reasoning with skill-based exploration, it paves the way for scalable, cross-task robot learning.