ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

Overview§

ADEPT introduces a framework for learning dexterous manipulation policies that transfer from simulation to reality across high-DoF robot hands. The key idea is to pretrain a single general dexterous policy on a generic object reposing task, then post-train it for specific downstream tasks. This avoids learning low-level manipulation skills from scratch for each new task. The authors address the challenge of catastrophic forgetting during fine-tuning via a carefully engineered post-training recipe, and they introduce a joint-space Geometric Fabric to mediate between the RL policy and the robot, enabling safe exploitation of the full kinematic dexterity.

Core Methodology§

Pretraining on Object Reposing§

The pretraining task requires the robot to reposition a generic object to a random target pose. This is formulated as a goal-conditioned RL problem. Let $s_t$ be the state (visuo-tactile observations + proprioception), $a_t$ be the joint-space action, and $g$ be the goal pose. The policy $\pi_\theta(a_t | s_t, g)$ is trained to maximize $\mathbb{E}[\sum_t \gamma^t r(s_t, a_t, g)]$, where the reward is a sparse or dense function of the distance between current and target object pose.

Post-Training Recipe§

Naive fine-tuning quickly degrades the zero-shot reposing capability. To prevent this, ADEPT combines:

1. Behavior-Cloning Distillation: A frozen pretrained policy acts as a teacher. During post-training, the student policy is regularized to match the teacher's actions on states relevant to reposing. This is implemented with an auxiliary loss:

$$ \mathcal{L}_{BC} = \mathbb{E}_{s \sim \mathcal{D}} \| \pi_\theta(s, g) - \pi_{\text{teacher}}(s, g) \|^2 $$

1. Critic Warm-Up: The critic is initialized from the pretrained critic and warmed up with a few gradient steps on the pretraining distribution before updating the actor on the new task.
2. Conservative On-Policy Updates: The policy updates are constrained to be close to the pretrained policy, e.g., by clipping the probability ratio in PPO or adding a KL penalty:

$$ \mathcal{L}_{KL} = \beta \, D_{KL}(\pi_\theta \| \pi_{\text{teacher}}) $$

Geometric Fabric§

To handle the high-DoF joint space and avoid unsafe movements, ADEPT introduces a Geometric Fabric—a task-space impedance controller that maps the RL policy's desired joint accelerations to torques while respecting geometric constraints (e.g., joint limits, collision avoidance). The fabric acts as a low-level filter, allowing the policy to operate in a smoother latent action space. This is analogous to using a residual policy on top of a fixed controller.

Sim-to-Real Transfer§

Policies are distilled into perceptive students that take raw RGB and tactile sensor images as input. The student is trained via DAgger-style imitation learning on rollouts from the teacher in simulation, and then directly deployed on real robots. The authors demonstrate zero-shot sim-to-real transfer on two embodiments: a 23-DoF Kuka-Allegro hand and a 29-DoF Flexiv-Sharpa hand, both solving long-horizon tasks (e.g., object reorientation, tool use) at human-level speed.

Implementation Sketch§

Below is a simplified PyTorch-style pseudocode for the post-training loop:

# Assume pretrained actor and critic are loaded
actor = Actor(obs_dim, action_dim)
critic = Critic(obs_dim + goal_dim)
actor.load_state_dict(pretrained_actor)
critic.load_state_dict(pretrained_critic)

# Post-training loop
for iteration in range(num_iters):
    # Collect on-policy data for downstream task
    trajectories = collect_rollouts(actor, env, task_goals)
    
    # Critic warm-up: only train critic on old and new data
    if iteration < warmup_steps:
        for batch in trajectories:
            loss = mse(critic(batch), batch.returns)
            loss.backward()
            optimizer_critic.step()
        continue
    
    # PPO update with BC distillation and KL penalty
    for batch in trajectories:
        # Compute KL to pretrained teacher
        kl_penalty = beta * kl_divergence(actor, teacher, batch.states)
        # Behavior cloning loss on reposing states
        bc_loss = lambda_bc * mse(actor(batch.states), teacher(batch.states))
        # PPO surrogate loss (clipped)
        ratio = (actor.log_prob(batch.actions) - batch.old_log_prob).exp()
        policy_loss = -torch.min(ratio * batch.advantages, 
                                 clip(ratio, 1-eps, 1+eps) * batch.advantages)
        total_loss = policy_loss + bc_loss + kl_penalty
        total_loss.backward()
        optimizer_actor.step()

Results & Takeaways§

ADEPT demonstrates that a single pretrained dexterous manipulation policy can serve as a reusable prior for various downstream tasks, drastically reducing the sample complexity and training time. The combination of behavior cloning distillation and conservative updates preserves the pretrained reposing ability while enabling new skills. The Geometric Fabric is crucial for transferring raw joint-space control to real robots without damaging hardware. This work provides a strong template for large-scale RL pretraining in robotics, similar in spirit to pretraining in NLP but adapted to continuous control with visuo-tactile inputs.