Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation

Core Problem§

Coding agents (LLM writes robot controller as a program, robot executes it) now operate robots without robot-specific training. But the paradigm's safety has been unexamined. Under a paired constraint—a manipulation goal plus an obstacle the robot must not touch—the agent collides in most cases. The failure is not perception (the model reasons about the obstacle in its traces) nor instruction (the prompt already forbids touching it); it is planning: the stated constraint never becomes a priority.

Where It Breaks§

The authors decompose manipulation into two phases:

1. Route phase — the model has no notion of a clearing route, and cannot replan once a chosen route becomes infeasible.
2. Contact phase — the model is unaware that contact execution is bounded by the same safety constraint.

Formally, given goal $g$, obstacle $O$, and start $s$, the agent solves $\pi = \arg\max_\pi P(g \mid \pi)$ but omits the feasibility constraint $\text{path}(\pi) \cap O = \varnothing$. Safety is never a term in the objective.

SafeHarness§

SafeHarness equips the model with two obstacle-aware harnesses:

1. Obstacle-Aware Route Planning§

• Grounds objects as bounding boxes $B_i$.
• Draws candidate routes as sequences of waypoints $w_1, \dots, w_n$.
• Agent plans a route in advance, verifies it against $B_i$, replans when the route is infeasible, and only then executes. This injects a feasibility check into the objective: $\pi^* = \arg\max_\pi P(g \mid \pi)$ s.t. $\forall i,\ w_i \notin B_i$.

2. Obstacle-Aware Contact Execution§

• Selects the contact position so the contact itself avoids $O$.
• Encodes the constraint at the moment of contact rather than only along the route.

Implementation Sketch§

import torch

def safe_route_plan(agent, goal, obstacles, start, max_replans=5):
    boxes = [o.bounding_box for o in obstacles]          # B_i
    for _ in range(max_replans):
        waypoints = agent.plan_waypoints(goal, boxes, start)  # w_1..w_n
        if all(not inside(w, b) for w in waypoints for b in boxes):
            return waypoints                              # feasible -> execute
        start = waypoints[-1]                             # replan from failure
    return None

def obstacle_aware_contact(agent, waypoints, obstacle):
    # choose contact pose c s.t. c not inside obstacle box
    c = agent.select_contact(waypoints)
    assert not inside(c, obstacle.bounding_box)
    return c

Results§

Relative to the same agent without harnesses, these are $2.3\times$ and $1.5\times$. The delta shows that most of the safety comes from making the constraint a first-class planning term—verification and replanning—rather than from better perception or stronger instructions.