Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

Core Methodology§

RPG (Reconstruct, Practice, Go Real) is a framework for autonomous improvement of robot execution systems without updating model weights. It operates in three phases:

1. Reconstruct: Identify manipulation capabilities from an offline dataset and construct related practice tasks in simulation.
2. Practice: Execute tasks in simulation, collect execution feedback, privileged simulator state, and dataset videos to diagnose failures. Based on diagnoses, RPG:

• Develops new reusable symbolic skills
• Refines existing skills
• Revises the system prompt
• Tests individual candidate changes and merged revisions via cross-task evaluation before retention.

1. Go Real: Deploy the frozen system (LLM + skill library + system prompt) to physical robot.

At test time, a multimodal LLM uses the resulting system prompt and skill library to coordinate perception and robot control. No model weights are updated; improvement is entirely in the symbolic skill library and prompt.

Technical Breakdown§

Skill Representation§

Skills are symbolic programs composed of perception primitives (e.g., detect object) and control primitives (e.g., move to pose, grasp). Formally, a skill $s$ is a sequence of actions $a_1, a_2, ..., a_n$ where each $a_i$ is drawn from a set of primitive actions $\mathcal{A}$. The skill library $\mathcal{S}$ is a set of such skills.

Practice Loop§

At round $t$, the system executes a task $\tau$ with current system prompt $P_t$ and skill library $\mathcal{S}_t$. The execution produces a trajectory $\xi = (o_1, a_1, o_2, a_2, ..., o_T, a_T)$ where $o_i$ are observations (images, proprioception) and $a_i$ are actions. Privileged simulator state $\zeta$ (e.g., object poses, contact forces) is also recorded. A diagnostic function $D$ analyzes $(\xi, \zeta)$ and any available dataset video $v$ to produce a diagnosis $d$:

$$d = D(\xi, \zeta, v)$$

The diagnosis $d$ is then used to update the skill library and prompt:

$$\mathcal{S}_{t+1}, P_{t+1} = U(\mathcal{S}_t, P_t, d)$$

where $U$ is an update function that may add new skills, modify existing skills, or change the prompt.

Cross-Task Evaluation§

Candidate changes (new skills, prompt edits) are evaluated on a set of held-out tasks $\mathcal{T}_{\text{eval}}$. A change is retained if it improves success rate on $\mathcal{T}_{\text{eval}}$ without degrading other tasks. This is a form of greedy search over the space of skills and prompts.

Implementation Details§

• The multimodal LLM (e.g., GPT-4V or similar) is used as the central controller. It receives the system prompt, skill library, current observation, and task description, and outputs a skill call or primitive action.
• Simulation environment: likely a standard robotics simulator (e.g., MuJoCo, Isaac Gym) with privileged state access.
• Dataset videos: used as reference for diagnosis (e.g., comparing failure modes).
• Calibration and hardware adaptation: a procedure to align simulation and reality (e.g., camera calibration, robot parameter tuning) before physical trials.

Code Snippet (Illustrative)§

class RPGAgent:
    def __init__(self, llm, skill_library, system_prompt):
        self.llm = llm
        self.skills = skill_library
        self.prompt = system_prompt

    def execute_task(self, task, sim_env):
        trajectory = []
        obs = sim_env.reset(task)
        while not sim_env.done:
            action = self.llm.predict(self.prompt, self.skills, obs, task)
            obs, reward, done, info = sim_env.step(action)
            trajectory.append((obs, action, info['privileged_state']))
        return trajectory, sim_env.success

    def diagnose(self, trajectory, dataset_video=None):
        # Use LLM to analyze failures and propose changes
        diagnosis = self.llm.analyze(trajectory, dataset_video)
        return diagnosis

    def update(self, diagnosis, eval_tasks):
        candidates = self.llm.propose_updates(diagnosis)
        for cand in candidates:
            if self.evaluate(cand, eval_tasks) > self.evaluate_current(eval_tasks):
                self.skills, self.prompt = cand.apply(self.skills, self.prompt)

    def evaluate(self, candidate, tasks):
        # Cross-task evaluation
        success = 0
        for task in tasks:
            traj, succ = self.execute_task(task, sim_env)
            success += succ
        return success / len(tasks)

Results§

• On 22 manipulation tasks with held-out initializations, RPG improves success from 28.6% after 1 practice round to 95.0% after 15 rounds.
• Outperforms baselines: ASPIRE (75.5%) and CaP-Agent0 with GPT-6 Astra Pro (60.0%).
• After calibration and hardware adaptation, the frozen system achieves 100% success on 30 physical trials (10 each on 3 tasks).

Key Insights§

• No weight updates: Improvement is purely symbolic, making it sample-efficient and interpretable.
• Sim-to-real transfer: Practice in simulation with privileged information enables effective real-world deployment.
• Self-improvement loop: Automatically generates and refines skills, reducing human effort.
• Cross-task evaluation prevents overfitting to specific tasks.

Limitations§

• Relies on a simulator with privileged state and a dataset of videos.
• Requires a powerful multimodal LLM for diagnosis and control.
• Calibration and hardware adaptation are still needed for physical deployment, though the procedure is common.